Why it matters
An agent without verification is a text generator that has touched your repository. It will report success, because the most likely continuation after "I have fixed the bug" is a summary of what was done. That report is not evidence, and treating it as evidence is how unverified changes reach production with a clean-looking transcript.
A verification loop closes the gap: the harness runs something external, observes the result, and only then allows the task to be marked done. The engineering question is not "does the agent test its work" but "what can fail, and would we see it".
Mental model
A smoke detector, not a fire report. The detector is independent of the fire, it triggers on a physical signal, and it is tested periodically. A report written by the person who may have started the fire is not a detector.
The analogy breaks on coverage. A smoke detector covers one hazard in one room. A test suite covers the cases someone wrote, and an agent can produce code that passes all of them while being wrong in a way nobody tested. Verification reduces uncertainty; it does not establish correctness.
How it works
Keyed transitions
- Widen → No tests: collected = 0
- Parse → Oracle edited: diff guard: tests edited
- Target → Edit: fail: feed back
Event · step 1 of 7
Edit staged
Entry: a diff exists and the task's done condition names the command to run. Exit is the parse stage, before any test is selected.
In state
- Edit
Next: Parse
- Checks run
- 0
- Collected
- —
- Verified run
- 0
Figure. Seven states. Each rung runs the next cheapest check and stops at the first failure, which loops back to an edit. Verified is only reachable through a recorded run; an empty test selection and an edited oracle are failure states, and the dotted edge shows that zero tests cannot be read as green.
What to notice. The harness runs the commands; the model never composes the verdict. The parse state covers the compile and type-check rungs: both are static, both must exit zero, and the targeted test only runs after them. The first non-zero exit stops the ladder. Commands, test counts and durations are illustrative; flaky-test quarantine, parallel shards and CI configuration drift are omitted.
Text version
A state machine with seven states. Edit is the start and runs Parse. Parse goes to Target on syntax and types ok, to Oracle edited on a diff that touches test paths, and back to Edit when compile or type checking fails. Target goes to Widen when at least one test was collected and the exit code is zero. Target and Widen loop back to Edit on a failed run, and either stage exits to No tests when its collected count is zero. Widen goes to Verified after the package and full suite pass and a workspace hash is recorded. No tests has a dotted, arrowless edge to Verified labelled zero tests, refused. The trace walks Edit, Parse, Target, Widen, Verified, then No tests, then Oracle edited. Counters show up to five checks run, 323 tests collected and one verified run.
- Define done before the work starts. "The endpoint returns 404 for a missing id, and the existing suite passes" is verifiable. "Add pagination" is not.
- Order the oracles by cost and strength. Compile or parse, then type check, then the targeted test, then the surrounding package, then the full suite. Stop at the first failure and report it precisely.
- Run narrow first. A single test file gives feedback in seconds; the full suite may take minutes. The agent's repair loop should operate on the fast oracle.
- Widen before declaring done. Passing one test does not mean the package builds. The final gate should be the project's real check, the same one CI runs.
- Make the oracle independent. Execute a real command. Do not ask the model whether the change looks correct; do not let it author both the code and the assertion in the same turn without review.
- Protect the oracle. Test files, fixtures and CI configuration are write-protected unless the task is explicitly about tests. An agent that can edit the assertion can always pass.
- Record evidence. Store command, exit code, collected test count, duration and a hash of the workspace state. "Verified" without a record is a claim.
- Feed failures back verbatim. The assertion diff and the relevant stack frames, truncated at the boundary. Paraphrase loses the detail that enables the repair.
CHECKS = [
("parse", ["python", "-m", "compileall", "-q", "src"], 60),
("types", ["mypy", "--no-error-summary", "src"], 120),
("target", ["pytest", "-q", "-x", "{target}"], 180),
("package", ["pytest", "-q", "tests/unit"], 600),
("full", ["pytest", "-q"], 1800),
]
def verify(ws, target, stop_on_failure=True):
evidence = []
for name, cmd, timeout in CHECKS:
argv = [part.format(target=target) for part in cmd]
result = ws.run(argv, timeout=timeout)
collected = parse_collected(result.stdout) # None when not a test run
evidence.append({
"check": name,
"cmd": argv,
"exit_code": result.code,
"tests_collected": collected,
"duration_s": result.duration,
"workspace_hash": ws.hash(),
})
if result.code != 0 and stop_on_failure:
return {"ok": False, "failed": name, "evidence": evidence}
if collected == 0 and name in {"target", "package", "full"}:
return {"ok": False, "failed": name, "reason": "no tests collected", "evidence": evidence}
return {"ok": True, "evidence": evidence}The tests_collected == 0 branch catches a specific and common false positive: the filter matched nothing, the runner exited zero, and the agent reported success. Treating an empty test selection as a failure is one of the highest-value lines in a harness.
Trade-offs
| Verification level | Good for | Cost | Fails when |
|---|---|---|---|
| Compile or parse only | Fast syntax repair | No semantic signal | The bug is behavioural |
| Targeted test | Inner repair loop | Narrow coverage | The change breaks a neighbour |
| Full suite in CI | Merge gate | Minutes per iteration | The suite is slow, flaky or incomplete |
| Property-based or fuzz tests | Parsers, invariants, edge cases | Setup effort, runtime | Properties are hard to state |
| Model-based review | Style, intent, missing cases | Unreliable, correlated with the author | It is the only check |
Good for
Fast syntax repair
Cost
No semantic signal
Fails when
The bug is behavioural
- The bug is behavioural
- The change breaks a neighbour
- The suite is slow, flaky or incomplete
- Properties are hard to state
- It is the only check
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Compile or parse only is good for Fast syntax repair, costs No semantic signal, and fails when The bug is behavioural. Targeted test is good for Inner repair loop, costs Narrow coverage, and fails when The change breaks a neighbour. Full suite in CI is good for Merge gate, costs Minutes per iteration, and fails when The suite is slow, flaky or incomplete. Property-based or fuzz tests is good for Parsers, invariants, edge cases, costs Setup effort, runtime, and fails when Properties are hard to state. Model-based review is good for Style, intent, missing cases, costs Unreliable, correlated with the author, and fails when It is the only check
Failure modes
- Empty test selection. The command exits zero because nothing matched. Debug: parse the collected count and fail when it is zero; log the filter expression in the evidence.
- Oracle tampering. The agent edits the failing test to match its implementation. Debug: protect test paths with a policy rule and flag any diff that touches them for human review.
- Flake-driven repair. A flaky test fails, the agent "fixes" unrelated code, and the next run passes by luck. Debug: track per-test failure history and quarantine known flakes before the agent sees them.
- Green build, wrong behaviour. Every check passes and the feature does not work. Debug: add one end-to-end assertion for the user-visible behaviour, and review the diff for what the tests do not cover.
In production
Feedback time is the operative metric. If the targeted check takes ninety seconds, the agent's repair loop is slow and expensive; if the full suite takes twenty minutes, nobody runs it before review. Invest in test selection and parallelism before investing in a bigger model.
Report verification coverage as a first-class number: the share of completed tasks with a recorded, non-empty, passing check. Pair it with escaped-defect rate to see whether the coverage is meaningful. A high coverage number with a high escape rate means the oracles are weak, not that the agent is unlucky.
Also budget the verification itself. Full-suite runs consume CPU and wall clock across every agent session; a fleet of twenty agents each running the suite on every turn is a capacity plan, not a detail.
Common misconceptions
People think: Asking the model to double-check its own work is verification.
Actually: It is a second sample from the same distribution, with the same blind spots and the same context. It catches some slips and cannot catch a shared misunderstanding.
People think: A passing test suite proves the change is correct.
Actually: It proves the tested cases pass. Correctness relative to the requirement still needs a reviewed diff and at least one assertion that encodes the requirement.
Interview angle
The question is often "how do you know the agent's change is good?" Answer with the oracle hierarchy and the independence requirement, then name the two cheapest wins: a fast targeted check and rejecting empty test selection. Follow-ups probe flaky tests and test tampering, both of which are policy problems rather than model problems. See agentic coding interview questions, TDD loops and trajectory evaluation.
Sources
- Anthropic, Claude Code best practices — instructs the agent to run checks and show evidence rather than assert success.
- Jimenez et al., SWE-bench — execution-based evaluation of patches against real test suites, and its limits.
- Xia et al., Agentless — patch validation and regression tests as the filter between generation and submission.
- pytest, Usage and invocation — selection, exit codes and collection semantics used by harnesses.