AI coding productivity is often sold as an obvious gain: give a developer a capable assistant and the same work should take less time. A randomised trial by the research organisation METR found the opposite in one demanding setting. Sixteen experienced open-source developers completed 246 real issues from repositories they already knew, and they took 19 per cent longer when AI tools were allowed. The surprise became sharper after the work was finished, because the developers still believed the tools had made them 20 per cent faster.
That finding does not prove AI slows down every programmer. It does show why confidence, adoption and benchmark scores are not substitutes for measuring completed work. The trial covered experienced maintainers working on mature, high-quality projects in early 2025, mainly with Cursor Pro and Claude 3.5 or 3.7 Sonnet. It is a narrow result, but it is narrow in a useful way: the researchers measured the sort of contextual, review-ready work that polished coding demonstrations often leave out.
What the AI Coding Productivity Trial Measured
METR recruited developers from large open-source repositories averaging more than 22,000 GitHub stars and more than one million lines of code. Each participant had contributed to their chosen repository for several years. The developers proposed issues that the project genuinely needed, including bug fixes, features and refactors, before those tasks were randomly assigned to an AI-allowed or AI-disallowed condition.
The design matters. Participants were not solving isolated puzzles written for a benchmark. They were navigating codebases full of conventions, undocumented assumptions, test requirements and maintainers’ expectations. They recorded their screens and reported the implementation time needed for each task. When AI was permitted, they could use whichever tools they preferred, although most used Cursor Pro with Anthropic models that were near the frontier at the time.
The original METR report says developers forecast a 24 per cent speed-up before doing the work. Instead, the measured result was a 19 per cent slowdown. The associated research paper reports robustness checks across different estimators and outcome measures, while also warning that sixteen developers cannot represent the whole software industry.
Why AI Added Time Even When It Felt Helpful
AI assistance changes the shape of coding work. Producing a plausible patch may take seconds, but deciding whether that patch belongs in a mature repository can take much longer. The developer has to explain local architecture to the model, inspect generated code, notice missed requirements, correct style problems, run tests and recover when the assistant follows an attractive but wrong path. The typing gets faster while the supervision expands.
This helps explain the gap between perception and measurement. A developer can feel rapid progress because code appears immediately and tedious searching is reduced. The hidden cost arrives in small review loops, interruptions and discarded suggestions. Those fragments are hard to remember at the end of a task, especially when the interaction itself feels easier. LiveAIWire’s earlier examination of AI productivity tools and lost focus describes the same accounting problem at workplace scale: visible output can rise while attention is repeatedly broken.
The trial also involved unusually experienced contributors. Their advantage was not merely typing speed. It was a dense mental model of the codebase and its unwritten rules. An assistant that lacks that context can force the expert to translate knowledge they would otherwise apply almost automatically. By contrast, a beginner, a developer entering an unfamiliar repository or someone producing disposable prototype code may gain far more from generated explanations and scaffolding.
Why the Result Does Not Settle the AI Coding Debate
METR explicitly says its study does not establish that AI fails to speed up most developers, other professions or even the same developers with later tools. The tasks averaged about two hours and came from large open-source projects with high quality standards. Agentic systems that can search more broadly, run longer and attempt several approaches may behave differently. So may work where the acceptable output is a quick internal script rather than a patch expected to survive public review.
Tool quality has also moved. In February 2026, METR published a follow-up on its later experiment. Raw results from late-2025 tools suggested possible speed-ups, including an estimated 18 per cent speed-up for returning developers, but the confidence interval crossed zero and the researchers judged the newer experiment unreliable. Developers who valued AI most were less willing to accept no-AI tasks, and participants avoided submitting tasks for which they expected especially large AI gains.
That update is not a reversal of the original trial. It is evidence that the effect is changing and becoming harder to measure. The early-2025 result remains a completed randomised test of a particular tool generation. The late-2025 data suggests newer systems may be more useful, while selection effects prevent a clean estimate of how much more useful. Both conclusions can be true at the same time.
Benchmark progress does not resolve the difference. A coding model can become much better at completing well-specified tasks while still imposing costs when a human must decide what the task really requires, integrate the change and satisfy standards the prompt never stated. Benchmarks are valuable measures of capability, but productivity is a property of the complete human and tool system. The same model can look exceptional on a test suite and add time inside a workflow built around tacit knowledge.
That is why repeatable field measurement matters. Teams can randomise comparable tickets, record active and elapsed time, review the accepted result and repeat the comparison as models and interfaces change. A single aggregate speed figure will still hide who benefits and which work benefits, but it is more informative than asking staff how much time they believe they saved after an enjoyable interaction.
AI Coding Productivity Depends on the Work Around the Model
The practical lesson is to measure the entire task, not the moment code appears. A team should compare elapsed time, accepted changes, review effort, defect rates and rework for comparable jobs. It should separate new prototypes from maintenance, familiar repositories from unfamiliar ones, and routine changes from work dominated by architecture or implicit product decisions.
This is also why simple model comparisons can mislead. LiveAIWire’s guide to ChatGPT and Claude for coding explains that the surrounding development environment, context access and review workflow often matter as much as the model. A strong model used through a poor interface can waste time. A weaker model embedded in a well-designed, tightly bounded workflow may save it.
Security and maintainability remain part of productivity too. A patch completed quickly is not a gain if it creates a vulnerability or becomes expensive to understand later. The risks documented in LiveAIWire’s coverage of AI-driven programming are not separate from the speed question. They are costs that appear after the stopwatch stops unless teams deliberately include them.
What Developers and Managers Should Take From the Trial
For developers, the result is permission to be selective. AI can be valuable for exploration, tests, documentation, unfamiliar APIs and repetitive transformations without belonging in every task. Turning it off when explaining the repository would take longer than doing the work is not resistance to technology. It is ordinary tool judgement.
For managers, licence adoption and employee enthusiasm are weak evidence of a productivity return. LiveAIWire’s analysis of AI workflow fatigue shows how quickly extra tools can become extra oversight. The useful question is not whether staff use AI, but where it reduces total cycle time without lowering quality or making expert review the new bottleneck.
The most durable conclusion from the METR trial is not that AI coding assistants are slow. It is that their value cannot be read from how fast they produce code or how productive they make someone feel. In one realistic early-2025 test, the promised acceleration turned into extra time, and the people doing the work did not notice. That is exactly the kind of result a serious productivity programme should be designed to find.
As tools improve, the number should be expected to change. The measurement standard should not. Claims of acceleration are strongest when they survive real work, complete quality checks and a comparison that includes every minute the human spends steering, waiting and repairing.
About the Author
Stuart Kerr is Technology Correspondent at LiveAIWire, covering artificial intelligence, cybersecurity and the social impact of emerging technology. LiveAIWire is an independent, human-led technology publication using AI-assisted research, editorial production and original AI-assisted editorial illustrations under his direction.