We have seen the introduction of several software developer productivity tools that leverage the latest advances in AI and LLMs to generate code. Let’s review initial conclusions on engineer productivity when using these services. We simply refer to productivity as one developer’s capability to generate code of the same or better quality, faster.
When Hugging Face and ServiceNow recently released another LLM tool that assists developers with generating software code, we thought it made sense to interpret the potential impact on the developer community’s productivity. At the end of the day, the tech sector overall, and more specifically, the tech services industry, leverage talent to write code, and these tools could eventually modify the relationship between the talent’s productivity and code quality, relative to the talent base size.
Some of the available code generating tools we found are listed below:
GitHub Copilot https://github.com/features/copilot
Amazon CodeWhisperer https://aws.amazon.com/codewhisperer/
BigCode StarCoder https://github.com/bigcode-project/starcoder
OpenAI’s Codex was deprecated and its features can be accessed directly via OpenAI’s Chat models, and Google DeepMind’s AlphaCode seems more of a one-off research project.
Intuitively, one could think that after an engineer gets used to relying on these tools, they would probably produce higher quality (fewer bugs) code, faster. There is some research that points to this being the case.
GitHub conducted research using a combination of surveys and experiments and found that developers using CoPilot feel more productive, and are happier than their control group not using CoPilot. Microsoft Research and GitHub documented their findings in a research paper which can be accessed here (https://arxiv.org/pdf/2302.06590.pdf). Results point to a ~55% increase in developer productivity.
Accenture used Amazon’s CodeWhisperer and shared overall improvements of 30% in developer productivity (https://aws.amazon.com/blogs/machine-learning/how-accenture-is-using-amazon-codewhisperer-to-improve-developer-productivity/).
In summary, although we are at the inception phase of these tools becoming mainstream, we start to see indications of better talent performance when individuals pair up their efforts with code generation tools. As this trend continues, within the next decade we believe the relationship between developer headcount and its relative productivity will continue to shift. We will be interested in finding the technology services companies that improve their margins and unit economics leveraging these tools.