Skip to content
AI Engineer Guide.

concept · coding · intermediate · concept 12 of 30

Patch validation

0 of 30 complete

Why it matters

An agent produces a diff as text, and text can apply cleanly while changing the wrong thing. The base commit may have moved, the diff may touch files nobody asked about, and the test the agent ran may not be the test that reproduces the bug. Validation turns those assumptions into mechanical checks, so review starts from evidence instead of a claim that the work is done.

Mental model

It is proofreading a translation against its source: same meaning, same places. The metaphor breaks because code has an executable oracle, so you can run the claim instead of only reading it, and the run can disagree with a diff that looks perfect.

How it works

Scope, base and the test that must failInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. A synthetic 20-line file at a recorded base with the line the patch replaces marked, its 3 + 1 + 3 hunk rectangle, and the same file after five lines were inserted: base row 8 now sits at row 13 and the unchanged rectangle covers the inserted rows. A repository map shows 3 files touched and 2 in the plan, and a test file has one assertion replaced. No commit, diff or applier was read.

What to notice. Applicability is spatial: the hunk must match exactly once, at the recorded revision, or the rectangle lands on other text. Scope is a set comparison: touched paths against the plan's files. Correctness is neither of those, and a deleted assertion makes the suite green while the specification shrinks.

Text version

A schematic file with a gutter and 20 lines, the changed line marked in accent. Steps show the base file, one hunk rectangle of three context, one changed and three context lines, the same file with five inserted rows and the base rows shifted down five so the rectangle covers the inserted text, a repository map with three files touched and two inside the plan, and a test file where one assertion line is deleted and a weaker one added.

  1. Apply the diff to a clean checkout of the recorded base commit and fail on any rejected hunk.
  2. Parse the diff to list touched files and assert they are a subset of the plan's scope.
  3. Run the test that reproduces the bug and require it to fail before the patch and pass after.
  4. Run the package test suite and the type check or build, stopping at the first failure.
  5. Reject whitespace-only churn, deleted assertions and unexplained lockfile or CI changes.
  6. Attach the command output to the patch so review sees evidence rather than a summary.

Trade-offs

ChoiceGood forCostFails when
Narrow reproduction testFast and targetedMisses collateral breakageChange touches shared code
Full suiteCatches regressionsSlow, flaky, expensiveSuite takes longer than the task
Strict scope checkPrevents drive-by editsRejects necessary adjacent fixesTask scope was under-specified
Agent-reported resultsNo harness to buildUnverifiable and easy to fakeCorrectness matters
Trade-off mapInteractive

Good for

Fast and targeted

Cost

Misses collateral breakage

Fails when

Change touches shared code

  1. Change touches shared code
  2. Suite takes longer than the task
  3. Task scope was under-specified
  4. Correctness matters

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Narrow reproduction test is good for Fast and targeted, costs Misses collateral breakage, and fails when Change touches shared code. Full suite is good for Catches regressions, costs Slow, flaky, expensive, and fails when Suite takes longer than the task. Strict scope check is good for Prevents drive-by edits, costs Rejects necessary adjacent fixes, and fails when Task scope was under-specified. Agent-reported results is good for No harness to build, costs Unverifiable and easy to fake, and fails when Correctness matters

Failure modes

  • Passing the wrong test. The patch makes an unrelated test pass while the reported bug remains. Debug: confirm the reproduction test fails on the base commit.
  • Context drift. The diff was generated against a branch that has since moved. Debug: store the base SHA in the patch metadata and apply against exactly that commit.
  • Deleted assertions. The agent removes a failing test instead of fixing the code. Debug: diff the test files and reject assertion deletions that lack a stated reason.

In production

Make validation a CI job on the patch rather than a step in the agent prompt. Track apply failures, scope violations and test-deletion attempts per model or harness change.

Common misconceptions

People think: If the tests pass, the patch is good.

Actually: Tests can pass while the bug persists if the reproduction was never added to the suite.

People think: A clean apply means the patch is compatible.

Actually: Textual application says nothing about semantic conflicts with concurrent changes.

Interview angle

Separate applicability, scope and correctness, then name the reproduction test that anchors all three. A strong answer stores the base revision, asserts the touched files and attaches command output as evidence. Equating a green suite with a correct patch, without asking whether the reproduction was added, is the weak version.

Sources