Developer Productivity with AI

We have seen the introduction of several software developer productivity tools that leverage the latest advances in AI and LLMs to generate code. Let’s review initial conclusions on engineer productivity when using these services. We simply refer to productivity as one developer’s capability to generate code of the same or better quality faster.

When Hugging Face and ServiceNow recently released another LLM tool that assists developers with generating software code, we thought it made sense to interpret the potential impact on the developer community’s productivity. At the end of the day, the tech sector overall, and more specifically, the tech services industry, leverages talent to write code, and these tools could eventually modify the relationship between the talent’s productivity and code quality relative to the talent base size.

Some of the available code-generating tools we found are listed below:

• GitHub Copilot https://github.com/features/copilot
• Amazon CodeWhisperer https://aws.amazon.com/codewhisperer/
• BigCode StarCoder https://github.com/bigcode-project/starcoder

OpenAI’s Codex was deprecated, and its features can be accessed directly via OpenAI’s Chat models. Google DeepMind’s AlphaCode seems more of a one-off research project.

Intuitively, one could think that after engineers get used to relying on these tools, they would probably produce higher-quality (fewer bugs) code faster. Some research points to this.

GitHub conducted research using a combination of surveys and experiments and found that developers using CoPilot feel more productive and happier than their control group that did not use CoPilot. Microsoft Research and GitHub documented their findings in a research paper, which can be accessed here (https://arxiv.org/pdf/2302.06590.pdf). Results point to a ~55% increase in developer productivity.

Accenture used Amazon’s CodeWhisperer and shared overall improvements of 30% in developer productivity (https://aws.amazon.com/blogs/machine-learning/how-accenture-is-using-amazon-codewhisperer-to-improve-developer-productivity/).

In summary, although we are at the inception phase of these tools becoming mainstream, we start to see indications of better talent performance when individuals pair up their efforts with code generation tools. As this trend continues, we believe the relationship between developer headcount and its relative productivity will continue to shift within the next decade. We are interested in finding technology services companies that can improve their margins and unit economics by leveraging these tools.