Why it matters
An agent produces a diff as text, and text can apply cleanly while changing the wrong thing. The base commit may have moved, the diff may touch files nobody asked about, and the test the agent ran may not be the test that reproduces the bug. Validation turns those assumptions into mechanical checks, so review starts from evidence instead of a claim that the work is done.
Mental model
It is proofreading a translation against its source: same meaning, same places. The metaphor breaks because code has an executable oracle, so you can run the claim instead of only reading it, and the run can disagree with a diff that looks perfect.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. A synthetic 20-line file at a recorded base with the line the patch replaces marked, its 3 + 1 + 3 hunk rectangle, and the same file after five lines were inserted: base row 8 now sits at row 13 and the unchanged rectangle covers the inserted rows. A repository map shows 3 files touched and 2 in the plan, and a test file has one assertion replaced. No commit, diff or applier was read.
What to notice. Applicability is spatial: the hunk must match exactly once, at the recorded revision, or the rectangle lands on other text. Scope is a set comparison: touched paths against the plan's files. Correctness is neither of those, and a deleted assertion makes the suite green while the specification shrinks.
Text version
A schematic file with a gutter and 20 lines, the changed line marked in accent. Steps show the base file, one hunk rectangle of three context, one changed and three context lines, the same file with five inserted rows and the base rows shifted down five so the rectangle covers the inserted text, a repository map with three files touched and two inside the plan, and a test file where one assertion line is deleted and a weaker one added.
- Apply the diff to a clean checkout of the recorded base commit and fail on any rejected hunk.
- Parse the diff to list touched files and assert they are a subset of the plan's scope.
- Run the test that reproduces the bug and require it to fail before the patch and pass after.
- Run the package test suite and the type check or build, stopping at the first failure.
- Reject whitespace-only churn, deleted assertions and unexplained lockfile or CI changes.
- Attach the command output to the patch so review sees evidence rather than a summary.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Narrow reproduction test | Fast and targeted | Misses collateral breakage | Change touches shared code |
| Full suite | Catches regressions | Slow, flaky, expensive | Suite takes longer than the task |
| Strict scope check | Prevents drive-by edits | Rejects necessary adjacent fixes | Task scope was under-specified |
| Agent-reported results | No harness to build | Unverifiable and easy to fake | Correctness matters |
Good for
Fast and targeted
Cost
Misses collateral breakage
Fails when
Change touches shared code
- Change touches shared code
- Suite takes longer than the task
- Task scope was under-specified
- Correctness matters
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Narrow reproduction test is good for Fast and targeted, costs Misses collateral breakage, and fails when Change touches shared code. Full suite is good for Catches regressions, costs Slow, flaky, expensive, and fails when Suite takes longer than the task. Strict scope check is good for Prevents drive-by edits, costs Rejects necessary adjacent fixes, and fails when Task scope was under-specified. Agent-reported results is good for No harness to build, costs Unverifiable and easy to fake, and fails when Correctness matters
Failure modes
- Passing the wrong test. The patch makes an unrelated test pass while the reported bug remains. Debug: confirm the reproduction test fails on the base commit.
- Context drift. The diff was generated against a branch that has since moved. Debug: store the base SHA in the patch metadata and apply against exactly that commit.
- Deleted assertions. The agent removes a failing test instead of fixing the code. Debug: diff the test files and reject assertion deletions that lack a stated reason.
In production
Make validation a CI job on the patch rather than a step in the agent prompt. Track apply failures, scope violations and test-deletion attempts per model or harness change.
Common misconceptions
People think: If the tests pass, the patch is good.
Actually: Tests can pass while the bug persists if the reproduction was never added to the suite.
People think: A clean apply means the patch is compatible.
Actually: Textual application says nothing about semantic conflicts with concurrent changes.
Interview angle
Separate applicability, scope and correctness, then name the reproduction test that anchors all three. A strong answer stores the base revision, asserts the touched files and attaches command output as evidence. Equating a green suite with a correct patch, without asking whether the reproduction was added, is the weak version.
Sources
- Jimenez et al., SWE-bench — hidden tests as the oracle for a generated patch.
- Git, git-apply documentation — how a diff is applied and where it can fail.
- Xia et al., Agentless — localise, repair and validate as separate stages.