Why it matters
Agent adoption is justified with a number, and the easiest numbers measure activity: suggestions accepted, lines generated, tasks attempted. None of them says whether the team shipped more value or spent more time reviewing. Productivity is multi-dimensional, and the dimensions move in opposite directions, so a measurement that captures only one of them will report a gain even when the net effect is a loss.
Mental model
It is a factory output gauge that ignores rework. The metaphor breaks because software's rework happens in review and incidents, far from the gauge, and the people reading the gauge usually chose what it measures and never reset it between shifts.
How it works
Step 1 of 8
Name the outcome and the class
Cycle time and change failure rate for one class of change. Adoption is a change to the delivery system, so the unit is a shipped change, not a suggestion or a line.
root · current
Outcome · delivered change, one class
One class of change, defined before adoption: a billing bug fix, a dependency bump, a new endpoint. A number without a class mixes work of different difficulty and calls the mix a gain.
Root of the hierarchy
Figure. Delivered value decomposed into speed, review load, quality and developer experience, each with the instrument that measures it. The crossed branch is acceptance rate used alone: it rises with volume, so it reports a gain while the two signals that could disagree go unmeasured.
What to notice. No numbers are drawn: the page publishes none, and a productivity claim without a comparison, a window and a change class is exactly what the page warns against. The four signals are the page's dimensions; the instruments named under them are examples of each. Omitted: the study design and its statistics, which the page covers as a before period, a matched team or an alternating policy.
Text version
A productivity measurement tree. The root is delivered change in one class, measured by cycle time and change failure rate. Four signal branches hang from it. Speed carries cycle time and lead time, with a crossed acceptance-rate proxy. Review load carries review wait per diff and follow-up fixes within two weeks. Quality carries defects by class and a gate requiring that an independent check actually ran. Developer experience carries a repeated load-and-trust survey, with a crossed first-month gain as the novelty confounder. Two more crossed branches sit at the root: one number with no comparison, and acceptance rate as the pitch. A note records how to claim a gain with a before period, a matched team or an alternating policy. Steps name the outcome, walk the four signals, rule out the proxies, and close on the comparison.
- Define the outcome first, such as cycle time for a class of change or change failure rate.
- Pick a comparison: a before period, a matched team or an alternating policy.
- Instrument the delivery pipeline using commits, reviews, deploys and incidents rather than individual activity.
- Measure review time and rework on agent-assisted changes separately from human-only ones.
- Survey developers for cognitive load and trust, and keep the survey as a separate signal.
- Report uncertainty and the confounders you could not control.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Task time in a controlled study | Causal clarity | Artificial tasks, small samples | Work is long-running and collaborative |
| Repository throughput metrics | Real code, large sample | Confounded by team and project mix | Teams differ in maturity |
| Developer survey | Captures load and trust | Self-report bias | You need a causal claim |
| A/B by team | Operational realism | Contamination between groups | Tool is available to everyone |
Good for
Causal clarity
Cost
Artificial tasks, small samples
Fails when
Work is long-running and collaborative
- Work is long-running and collaborative
- Teams differ in maturity
- You need a causal claim
- Tool is available to everyone
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Task time in a controlled study is good for Causal clarity, costs Artificial tasks, small samples, and fails when Work is long-running and collaborative. Repository throughput metrics is good for Real code, large sample, costs Confounded by team and project mix, and fails when Teams differ in maturity. Developer survey is good for Captures load and trust, costs Self-report bias, and fails when You need a causal claim. A/B by team is good for Operational realism, costs Contamination between groups, and fails when Tool is available to everyone
Failure modes
- Proxy fixation. Acceptance rate rises while review time doubles. Debug: plot review latency and rework against the adoption date.
- Selection effect. Only easy tasks are delegated and the average looks better. Debug: compare the task difficulty distribution before and after.
- Novelty effect. Short studies show gains that fade. Debug: extend the measurement window past the novelty period.
In production
Track cycle time, review latency, change failure rate and revert rate together. Re-baseline after a model or harness change so old comparisons are not reused.
Common misconceptions
People think: Agent adoption can be evaluated with one number.
Actually: Speed, quality and load move independently, so a single metric hides two of them.
People think: Developers are the best judge of their own speedup.
Actually: Self-report captures perceived effort, which diverges from measured time on real tasks.
Interview angle
Name the outcome, the comparison and the confounders before quoting any number. A strong answer reports speed alongside quality and load and explains why acceptance rate is an activity metric. Repeating a vendor percentage, with no task mix or review cost, is the weak version.
Sources
- METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity — a controlled study where measured time diverged from expectation.
- DORA, Research program — delivery metrics such as lead time and change failure rate.
- Forsgren et al., The SPACE of developer productivity — a multi-dimensional framework for productivity.