Why it matters
Agents optimise for output that looks plausible, and plausible code can satisfy a prompt while missing the actual defect. Test-first development supplies an oracle before the implementation exists, which turns an open-ended generation into a search with a stopping condition. It also limits collateral edits, because any change that does not move the failing test is unnecessary by construction. Reviewing the test diff is part of the loop, because the agent controls both sides of the oracle and can weaken the assertion it is meant to satisfy.
Mental model
It is cutting a lock and key together: the test is the key and the code is the lock. The metaphor breaks because the agent can file the key down until any lock opens, which is why the test diff needs its own review rather than trust in the loop.
How it works
- 1Turn the report into one assertion about observable behaviour with the smallest input that triggers it.
- 2Run the test on the base commit and capture the failure message as evidence.
- 3Allow changes to production code only until that test passes, then stop the edit loop.
- 4Run the surrounding package tests to catch damage outside the targeted behaviour.
- 5Review the test diff as carefully as the code diff, looking for weakened assertions.
- 6Add a second test only for a distinct behaviour, never for another branch of the same one.
Step 1 of 6
Turn the report into one assertion about observable behaviour with the smallest input that triggers it.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Turn the report into one assertion about observable behaviour with the smallest input tha… 2. Run the test on the base commit and capture the failure message as evidence. 3. Allow changes to production code only until that test passes, then stop the edit loop. 4. Run the surrounding package tests to catch damage outside the targeted behaviour. 5. Review the test diff as carefully as the code diff, looking for weakened assertions. 6. Add a second test only for a distinct behaviour, never for another branch of the same one.
- Turn the report into one assertion about observable behaviour with the smallest input that triggers it.
- Run the test on the base commit and capture the failure message as evidence.
- Allow changes to production code only until that test passes, then stop the edit loop.
- Run the surrounding package tests to catch damage outside the targeted behaviour.
- Review the test diff as carefully as the code diff, looking for weakened assertions.
- Add a second test only for a distinct behaviour, never for another branch of the same one.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Strict test first | Clear stopping condition | Slow start when behaviour is hard to assert | Change is exploratory or UI-only |
| Characterisation test of current behaviour | Safe refactors | Locks in existing bugs | Current behaviour is itself wrong |
| Agent writes test and code | Fast | Self-confirming tests | Correctness matters |
| Snapshot tests | Cheap coverage | Brittle and reviewed as noise | Output is non-deterministic |
Good for
Clear stopping condition
Cost
Slow start when behaviour is hard to assert
Fails when
Change is exploratory or UI-only
- Change is exploratory or UI-only
- Current behaviour is itself wrong
- Correctness matters
- Output is non-deterministic
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Strict test first is good for Clear stopping condition, costs Slow start when behaviour is hard to assert, and fails when Change is exploratory or UI-only. Characterisation test of current behaviour is good for Safe refactors, costs Locks in existing bugs, and fails when Current behaviour is itself wrong. Agent writes test and code is good for Fast, costs Self-confirming tests, and fails when Correctness matters. Snapshot tests is good for Cheap coverage, costs Brittle and reviewed as noise, and fails when Output is non-deterministic
Failure modes
- Test written to match the patch. The assertion encodes the implementation including its bug. Debug: check out the base commit, apply only the test and confirm it fails.
- Assertion weakening. The agent loosens an existing assertion until it passes. Debug: diff test files and flag removed conditions or broadened matchers.
- Green on the wrong suite. The targeted test passes but the package suite was never run. Debug: record which command produced the green result and re-run the wider suite.
In production
Track tests added per task and mutation score on changed files. Reject patches that modify assertions without a recorded failing-first run.
Common misconceptions
People think: TDD means the agent writes tests, so the tests are trustworthy.
Actually: Tests written after the fix tend to encode the fix, so provenance matters more than presence.
People think: A passing suite means the feature works.
Actually: It means the assertions you have hold; coverage of the new behaviour is the open question.
Interview angle
Anchor the loop on a test that fails for the right reason before code changes. A strong answer reviews the test diff as carefully as the code and mentions mutation testing or assertion checks. Describing test-first as a prompt instruction, where the agent writes both sides and nobody inspects the assertions, is the weak answer.
Sources
- Kent Beck, Test-Driven Development by Example — the red, green, refactor loop.
- pytest documentation — fixtures, assertions and failure reporting.
- Chen et al., Teaching LLMs to Self-Debug — using execution feedback to repair code.