How to Tell If AI Is Actually Making Developers More Productive

Is AI actually making developers more productive?

No, not reliably, and often the opposite. The best controlled evidence we have says AI slows experienced developers down while making them feel faster. METR ran a randomized controlled trial: experienced open-source developers completed real tasks with and without AI tools. The group using AI took 19% longer to finish, not less. Before the study, they predicted AI would cut their time by 24%. After being measured as slower, they still walked away believing AI had sped them up by 20%. That gap, between what happened and what people believed happened, is the whole story of this article.

A year later, METR tried to run the experiment again and couldn't get a clean answer. So many developers now refuse to work without AI that researchers couldn't isolate a baseline. The result ranged from an 18% speedup to a 4% slowdown depending on the subgroup. That's not a productivity story. That's a measurement problem: adoption outrunning verification.

Google's DORA 2024 report, drawn from a large multi-year developer survey, found the same split personality at the team level. A 25% increase in AI adoption correlated with better documentation, better code quality, and faster reviews, but also an estimated 1.5% drop in delivery throughput and a 7.2% drop in delivery stability. DORA's explanation is blunt: AI removes the friction that used to force developers into small, reviewable batches of code. Cheap generation means bigger diffs, and bigger diffs are riskier to ship.

So if you're asking whether your AI stack is making your team more productive, the honest answer starts with a different question: productive at what? Writing code, or shipping and maintaining software that works? Here are five signs you're optimizing for the wrong one.

Are you shipping more code while delivery gets less stable?

This is the paradox DORA surfaced in aggregate. Faros AI's telemetry makes it granular, tracking 22,000 developers across 4,000 teams. Under high AI adoption:

• Pull request size rose 51%.
• Bugs per PR rose roughly 29%, with monthly incidents up 57.9% and bugs per developer up 54%.
• Median code review time rose roughly 5x.
• Incidents per PR rose 242.7%, close to a tripling.
• Code churn rose 861%, meaning code is rewritten and discarded almost as fast as it's written.

Every one of those numbers describes more motion, not more delivery. If your dashboards show commit volume and PR count climbing while cycle time to production and incident counts climb with them, that's not acceleration. That's a bigger, messier pile moving at the same speed, or slower.

Is your "done" pile full of prototypes that never reach production?

AI makes it trivially cheap to produce something that looks finished: a working demo, a clickable prototype, a pilot that impresses a stakeholder in a meeting. It does not make it cheap to produce something that survives real users, real load, and real maintenance.

A survey of 1,247 publicly launched "vibe coded" projects found only 3.4% were still active a year later. The causes of death line up with exactly the gap you'd expect between generating code and owning it:

1. Death by scaling, 34% of failures: the thing worked at demo scale and fell over at real scale.
2. Death by security, 28% of failures: holes nobody checked for because nobody owned the checking.
3. Death by maintenance, 21% of failures: no one could safely change the code six months in.

MIT NANDA's enterprise-scale research tells the same story at a bigger budget. About 95% of enterprise generative AI pilots fail to deliver measurable P&L impact, and only 5% of integrated pilots extract meaningful value. Gartner expects the agentic version to repeat: more than 40% of agentic AI projects will be canceled by the end of 2027, mostly from cost, unclear value, or weak risk controls rather than the model falling short. The common thread isn't model capability. It's that nobody built the muscle to turn a working prototype into owned, maintained software. If your team measures AI output in demos shipped rather than systems maintained, you're counting the wrong pile.

Are you spending more time reviewing AI code than you saved writing it?

The time AI saves generating a first draft has to go somewhere, and the data says it goes straight into review and rework. METR's trial measured developers 19% slower overall, even counting AI's generation speed. Faros measured review time rising roughly 5x under high adoption. That's not developers reading slowly. That's a review process absorbing bigger, riskier diffs than it was built for, the same mechanism DORA described.

GitClear's analysis of over 200 million lines of code between 2020 and 2024 shows what that reviewed code actually looks like once it lands. For the first time, copy-pasted code exceeded moved (refactored, reused) code in 2024. The share of changes tied to genuine refactoring fell from 25% of changed lines in 2021 to under 10% in 2024. AI isn't producing code that integrates cleanly into a codebase. It's producing code that gets pasted in and left there, which means the review time you're paying for today doesn't buy a cleaner codebase tomorrow. If your CI pipeline is choking on volume it was never built to validate, that bottleneck has a name, and teams are already restructuring their checks around it.

Does your team trust the output less the more of it there is?

Usage and trust are moving in opposite directions, and that gap is a tell on its own. Stack Overflow's 2025 Developer Survey, with more than 49,000 respondents, found AI tool usage at 84%. At the same time, 46% of developers said they actively distrust the accuracy of AI-generated code, up from 31% the year before, versus just 33% who trust it and only 3% who trust it highly. DORA found a similar split: 39% of respondents reported little to no trust in AI-generated code even as adoption climbed.

Rising usage paired with falling trust means the output isn't being verified. It's being accumulated. Teams use AI because it's fast and because not using it has become socially and competitively awkward, not because they've confirmed the results hold up. Trust should track with evidence. When it moves the opposite direction from adoption, the volume of code in your repo is outrunning anyone's confidence in what that code does.

Does nobody own what gets built because it's shadow AI, not shipped software?

Here's the part that never shows up in a sprint report. MIT NANDA's research into enterprise GenAI use uncovered what it calls a shadow AI economy: fewer than 40% of surveyed companies have official LLM subscriptions, but employees at more than 90% of companies routinely use personal AI accounts for work. A lot of what gets counted informally as "AI productivity" at the company level is software built by individuals, on tools IT never approved, with no one accountable for what happens when it breaks.

This is the same pattern documented across the tools employees are using to build their own apps: working software that nobody logged, nobody reviewed under a real process, and nobody is on the hook to maintain. It feels like output. It counts as output in nobody's budget, nobody's uptime guarantee, and nobody's incident response plan. Busywork dressed as initiative is still busywork, and it's the least visible version of the productivity illusion, because it never shows up in the DORA or Faros numbers above. Those measure sanctioned systems. The shadow stack is bigger and less accountable than any of them.

What is the real fix: measuring ownership and delivery instead of output volume?

Every sign above traces back to the same mistake: treating AI-generated artifacts as the product instead of treating shipped, maintained, trusted software as the product. Lines of code, PRs opened, prototypes demoed: none of that is revenue, reliability, or a system someone can safely change next quarter.

The fix isn't slower AI. It's a different scoreboard. Track what gets shipped to production and stays there. Track incident rate per release instead of releases per week. Track whether the person who built something is still the person who can maintain it six months later. That's why consolidating AI-assisted development onto tooling your team actually controls matters more than which model writes the fastest first draft. An AI-native workflow built around tools you own rather than a dozen unmanaged vendors keeps the speed without inheriting someone else's governance gap, exactly the gap MIT NANDA's shadow AI data describes.

The number that belongs on your dashboard isn't how much AI wrote this sprint. It's how much of what AI wrote is still running, still trusted, and still owned by someone with their name on it.