Intelligence Amplification in Code

The first working patch is an intermediate result. Somebody still has to decide whether it solves the right problem, fits the repository, and can be maintained. An AI tool can accelerate that whole process. It can also accelerate the first part while adding work to the rest.

That is why a claim such as “developers are 50% faster” needs a second sentence. Faster at what, using which tool, under what conditions, and measured to which finish line?

Two studies with different finish lines

In a controlled GitHub Copilot experiment reported in 2023, researchers recruited 95 professional programmers and randomly assigned access to Copilot. The task was to implement an HTTP server in JavaScript. Success meant passing twelve specified tests.

Among those who completed the task, average time was 71.17 minutes with access and 160.89 minutes without it: a 55.8% reduction in completion time, with a reported 95% confidence interval of 21% to 89%. The time result is conditional on completion. The estimated difference in completion rates was not statistically significant.

That is evidence for a bounded task with an observable finish line. It is not a measurement of months of maintenance, design review or production reliability. And the experiment ran in 2022; it is evidence about that tool and setting, not a current product comparison.

METR later studied experienced contributors working on real issues in repositories they knew well. Its early-2025 experiment involved 16 developers and 246 issues, randomly assigned to allow or disallow AI use. The work included the repository's testing, style and documentation expectations. With AI allowed, developers took 19% longer, with a reported confidence interval from 2% to 39% longer.

That result is historical. METR now explicitly warns that it should not be treated as the current impact of AI coding tools. Citing the slowdown without the update would be misleading.

The update makes the story less tidy

In its February 2026 follow-up, METR reported results from a later experiment and said the data were an unreliable signal of the current productivity effect. Developers increasingly declined participation or withheld tasks they did not want to do without AI. Compensation had also fallen, from $150 to $50 per hour. Concurrent agent use made time reporting harder.

The raw estimates pointed towards less time with AI: 18% less among returning developers and 4% less among newly recruited developers. Both confidence intervals included no effect. More importantly, the researchers thought selection effects obscured the population they wanted to measure. They believed developers were likely getting more benefit than before, but described the evidence for the size of that change as very weak.

We should neither freeze the earlier slowdown into a permanent law nor turn the later point estimates into a guaranteed speedup. The useful lesson is that task choice, developer selection and the meaning of “time spent” can change as the tools change.

Compare what the studies do and do not establish

Copilot experiment: randomised access, a specified JavaScript task, test-based completion. The reported time reduction is among completers. It does not price later maintenance.

Early METR experiment: randomised AI permission for selected repository issues with experienced maintainers. It does not represent every developer or newer tools.

Later METR data: broader participation and newer tools, but serious selection and measurement concerns. A more recent number is not automatically a more reliable population estimate.

These are separate comparisons. Averaging their percentages would produce a number without a coherent task, population or tool behind it.

Define what must survive the patch

For a hypothetical example, imagine adding CSV import validation to a small application. The assistant can propose parsing code, list edge cases and draft tests. Before generating changes, someone needs to settle the contract: which date formats are accepted, how duplicate rows behave, whether invalid input can partially change stored data, and what users should see when the import fails.

Those decisions need an accountable owner, not necessarily an owner who types every line. AI can help explore them. Treating architecture as a permanently human-only activity is as unhelpful as assuming generated implementation is automatically safe.

A compact task brief could specify the intended behaviour, a few disallowed outcomes, the files expected to change and the evidence required for acceptance. The brief does not guarantee a good patch. It gives the author and reviewer a shared basis for noticing a bad one.

Keep tests independent enough to challenge the implementation. If the same misunderstanding shapes both the generated parser and its generated tests, passing tests may only show that the two agree. Add cases derived from the contract, including a failure partway through an import and an input that should leave existing data untouched.

Review the scope too. A useful small fix can arrive wrapped in an unnecessary refactor. More generated code is more code to understand, even when it passes. But there is no universal line-count limit that makes a change reviewable; a short permission change may require more attention than a long mechanical rename.

Count the work that moves to somebody else

A sensible local evaluation would track accepted changes, review effort, rework and defects over a defined follow-up period. Keep active human time distinct from elapsed delivery time. If an agent runs while the developer does another task, counting every minute twice exaggerates labour cost; counting none of the review time exaggerates the saving.

Include abandoned attempts. If the hardest AI-assisted changes disappear from the sample, the remaining completion times can make the workflow look better than it is. Record tool versions and what assistance was permitted so a later result can be compared honestly.

And ask whether the quality bar changed. Extra documentation or better tests may be a worthwhile use of saved time even when delivery takes just as long. Conversely, fewer tests can make a speedup cheap to obtain and expensive to keep.

I would start with a narrow class of changes and keep the option to stop using the assistant there. A tool that helps with test scaffolding might be a poor fit for another task in the same repository. The decision should follow accepted work and total cost, not allegiance to either “AI writes everything” or “humans must write it all.”

The finish line is a change the team can stand behind. Measure to that line.

Sources and revision note

Peng and colleagues, The Impact of AI on Developer Productivity: Evidence from GitHub Copilot (2023), Study Design and Results. METR's July 2025 report and February 2026 update supply the repository-task evidence and its limits. This revision removes unsupported coordination-cost and rejection-rate generalisations, private system descriptions and invented causal breakdowns. The CSV workflow is hypothetical.