Does AI Slow Down Advanced Coders? by Dr. Timothy Smith

Photo Source: PIXNIO

The promise of AI-powered productivity gains has captivated industries worldwide, with software development at the forefront of this revolution. Tools like GitHub Copilot, Cursor Pro, and ChatGPT have been described as fundamental drivers of a new, more automated way of writing computer code. However, new research from the Model Evaluation & Threat Research (METR) group reveals a striking disconnect between perception and reality regarding AI's impact on the productivity of experienced developers. The not-for-profit group METR works to develop scientific methods to evaluate the potential risks posed by AI and methods to mitigate these threats.

In a study titled "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity" by Joel Becker, Nate Rush, Beth Barnes, and David Rein, the authors describe a comprehensive randomized controlled trial involving 16 experienced open-source developers completing 246 real-world computer programming tasks. (metr.org) The subjects of the study were given random problems to fix in the computer code and asked to predict how long it would take them to fix the code with or without the aid of AI. Then, the subjects would be randomly assigned to receive either AI or no AI assistance. They would then record the estimated time each task would take. These researchers found something remarkable —AI tools actually slowed down developers by 19%, despite widespread predictions of significant speedups. Before the study, developers forecasted that AI would reduce their completion time by 24%. Even after using AI tools extensively, they estimated a 20% speedup, while the data showed they worked significantly slower.

The expectation of AI to increase coding speed extends beyond the developers themselves. Machine learning experts predicted a 38% speedup, while economics experts forecasted 39% improvement. The unanimous expectation of dramatic productivity gains stood in stark contrast to the measured reality.

The METR study's detailed analysis suggests several reasons why AI tools might slow down experienced developers despite their impressive capabilities. Screen recordings showed that when using AI, developers spent less time actively coding and searching for information. Instead, they devoted significant time to prompting AI systems, waiting for responses, and reviewing AI-generated outputs, which required substantial cleanup in most cases. Without AI assistance, the code developers worked in a manner optimized to their experience and history working with large programs, which had over 1.1 million lines of code.

The authors suggest several reasons for the loss of efficiency when working with AI tools. They suggest that high developer expertise rendered the AI tools less helpful. The study focused on developers with an average of 5 years of experience on their respective computer programs. Their extensive domain knowledge made it challenging for AI to provide meaningful assistance, as the tools lacked a crucial understanding of context. Additionally, the authors concluded that the complexity of the computer programs that they worked on in the study caused the AI tools to struggle in these sophisticated environments, requiring extensive human oversight and correction. Finally, the study revealed that code writers accepted less than 44% of AI-generated code suggestions, and the majority reported making significant changes to clean up the AI outputs. This low reliability meant that AI assistance often created more work than it eliminated.

The METR study does not suggest that AI tools are inherently harmful or useless; however, it illustrates the importance of understanding when and how AI tools provide genuine value. The study's findings may not apply to junior developers working on simpler projects, where AI assistance could prove more beneficial.

As we navigate the artificial intelligence revolution, the goal should not be to maximize AI usage, but to optimize the human-AI partnership. Just as we learned to balance physical convenience with exercise after the Industrial Revolution, we must now learn to balance cognitive convenience with mental fitness in the age of AI. The future of productivity and prosperity lies not in complete abdication to AI but in thoughtful integration that preserves and enhances human creativity and cognitive abilities while leveraging AI's strengths appropriately.

Dr. Smith’s career in scientific and information research spans the areas of bioinformatics, artificial intelligence, toxicology, and chemistry. He has published a number of peer-reviewed scientific papers. He has worked over the past seventeen years developing advanced analytics, machine learning, and knowledge management tools to enable research and support high-level decision making. Tim completed his Ph.D. in Toxicology at Cornell University and a Bachelor of Science in chemistry from the University of Washington.

You can buy his book on Amazon in paperback and in kindle format here.