How to Measure AI Coding Agent Productivity in Your Team
Lines of code and PR counts miss what AI coding agents change. Here is how Weave models engineering work, tracks quality, and ties it to cost.
How do you measure whether AI coding agents make a team more productive?
Measure the work itself, not the activity around it. Lines of code and PR counts break down once agents write most of the code, because agents inflate both without proving value. A better approach reads the actual code changes, estimates the effort and complexity they represent, and tracks quality downstream: bugs, what happens at merge, and how the change holds up weeks later. Weave, a startup built around this idea, applies one measurement model to both human engineers and coding agents.
TL;DR
- Lines of code and PR counts were weak proxies for human output and get worse when agents generate code at volume.
- Weave's approach uses models that read the code and judge what the work was, rather than counting artifacts.
- The same measurement that worked for humans extended to agents, so teams can compare human and agent output on one scale.
- Speed numbers mean little without quality signals, such as bug rates and how changes look after merge.
- Early adopters hit cost questions first. Weave's founders say they sit roughly six months ahead of the typical curve because their customers are fast-growing startups.
- Once usage is measured, routing tasks to cheaper models becomes a testable cost lever instead of a gamble.
Why do lines of code and PR counts fail as metrics?
They count output that is easy to see, not output that matters. The founders of Weave describe the historical state of engineering measurement bluntly: you could count lines of code or number of pull requests, and those metrics mostly existed to make executives feel good.
The problems are familiar to any engineer:
- A large diff can be boilerplate, generated code, or a rename. A three-line fix can be the hardest change of the month.
- PR count rewards splitting work into small pieces, not finishing important work.
- Neither number says anything about bugs introduced or quality problems that surface later.
Agents make this worse. A coding agent can produce thousands of lines in minutes. If you measure volume, the agent always wins, and you learn nothing about whether the product got better.
How does Weave model engineering work?
Weave trains models to read the code and understand what an engineer (or agent) actually did. The founders point out that this only became possible with modern LLMs. Earlier tooling could count, but it could not read.
Reading the work lets a system estimate how much was accomplished and how many bugs were created, which is the part simple counters could never see. Weave's pitch is turning a software company into what it calls a "software factory": measure the work precisely enough that you can optimize how people and agents work, and how they work together.
The company says customers include Robinhood, PostHog, Telnyx, Extra Space Storage, and Reducto, ranging from tiny startups to Fortune 100 companies.
Why did human measurement carry over to coding agents?
The founders say the move was almost forced by customers. They joined Y Combinator's Winter 2025 batch while Cursor was starting to take off. Teams began writing much of their code with AI, and they asked a direct question: what is Cursor doing to our productivity?
If your model already evaluates the output of a code change, it does not matter much whether a person or an agent produced it. The same questions apply. How substantial was the change? Did it introduce defects? Did it survive review and production? That is why the human-focused work "could easily be applied to coding agents," in the founders' words.
The same period carried a second shift. In that batch, a significant number of founders reported that most of their code was being written by LLMs. That drew public pushback at the time. It now reads as an early signal.
What should you track besides speed?
Velocity alone is a trap. A useful measurement setup tracks at least two sides:
| Dimension | Example question |
|---|---|
| Velocity | How much meaningful work ships per person or per agent session? |
| Quality | Are bug rates or quality issues rising as agent usage rises? |
| Downstream effects | How does the change look in the PR, at merge, and a month later? |
| Cost | What did the output cost in tokens or subscription usage? |
Weave describes its own router as trained partly on these downstream signals, including how a session is reflected in the resulting PR, at merge to production, and a month later. The point carries beyond any vendor. If you only look at what happens inside the agent session, you miss whether the work was good.
What does the AI coding adoption curve look like?
The founders describe a curve where the newest, fastest-growing companies adopt coding agents first, and larger companies follow roughly six months to a year behind. Because Weave's early customers were startups living on the edge of agent use, the team says it was always 6 months ahead of the curve and saw problems before the wider market did.
The brief for this piece frames the progression as AI adoption, then "token maxing," then cost optimization. The interview supports that shape in general terms, though not as a formal framework:
- Adoption. Teams start using coding agents and want to know the impact.
- Token maxing. Heavy users push premium subscription plans as hard as they can, sending most work to the strongest model because the usage feels subsidized. One host describes being a "hardcore" user maxing out two $200 plans.
- Cost optimization. Once usage is high, teams look for ways to stretch limits. Not every task needs the biggest model.
Treat that as an observed pattern, not a law. Your team may skip stages or run them at once.
How does model routing fit into cost optimization?
Weave Router is the company's answer to stage three. It sits in front of a coding agent and picks a model per prompt or task. The founders say it works with CLI-based agents such as Claude Code, Codex, and Open Code, and can be plugged in with one command. They also say it is fully open source, with a hosted version and paid subscription plans.
Their reasoning is simple: "you don't always need a bazooka where a handgun will do." Exploring a codebase to find where auth lives, for example, can go to a cheaper model. Open source models such as DeepSeek, Kimi, and GLM, which the founders say performed best in their internal testing, handle simpler jobs like centering a button or writing a CRUD backend. Heavier work, such as building an ML system to classify a huge image set, should still go to a top-tier model.
On their own recent benchmarks, the founders claim frontier-level performance at half the cost and twice the speed. They also say subscription users can stretch the same spend about twice as far. These are vendor claims from an interview, and you should verify them on your own workloads.
How do you know cheaper routing doesn't hurt quality?
You measure it. This is the clearest argument for pairing any cost optimization with productivity measurement. The Weave team says it uses the router internally and sees its output continuing to accelerate, with no increase in bugs or quality issues, placing the team in the top 1% of teams in its own data.
That is a self-report, so treat it as a starting point. The practical lesson holds regardless: before and after you change models or routing, compare bug rates, review outcomes, and merged-change quality, not just token spend.
Frequently Asked Questions
Why not just count merged PRs?
PR count mixes together tiny and huge changes and can be gamed by splitting work. With agents, volume rises quickly without a matching rise in value. You need a measure that reads what each change actually does.
Can the same metrics apply to humans and agents?
Weave's experience says yes. Its models evaluate the code change itself, so the author does not matter. That lets a team compare human output, agent output, and mixed workflows on the same basis.
Is routing to open source models safe for production code?
It depends on the task. The founders say open source models like DeepSeek, Kimi, and GLM are not at the frontier but are close enough for simpler work, and that hard problems should go to stronger models. Check quality metrics on your own codebase before shifting traffic.
Does routing work with subscription plans?
According to the founders, yes. You can log in with multiple accounts, and the router spreads usage across subscriptions while mixing in open source models where they perform well. Confirm current plan behavior with the project's documentation.
How we source, write, and correct our articles. Editorial standards
