Why it matters
An agent works across many steps, and the only defensible outcome measure is whether the environment reached the required state. That state has to be asserted independently, because the agent's own report is text and text is exactly what the model produces well. Task success is the anchor metric for agents, and its credibility depends entirely on the checker behind it.
Mental model
It is a delivery confirmed by signature, not by the courier's message. The metaphor breaks because some goals have no signature, only a reasonable stopping point that someone has to define before the work starts, and that definition becomes part of the metric.
How it works
- 1Write the end-state assertion first: which row, file content or API response proves completion.
- 2Reset the environment to a known state before each attempt and record the reset hash.
- 3Run the checker independently of the agent's own test command.
- 4Record attempts, wall-clock time and tokens per success alongside the binary result.
- 5Add partial-credit assertions for intermediate state and report them separately.
- 6Audit the checker with a deliberately incomplete solution to confirm it fails.
Step 1 of 6
Write the end-state assertion first: which row, file content or API response proves completion.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Write the end-state assertion first: which row, file content or API response proves compl… 2. Reset the environment to a known state before each attempt and record the reset hash. 3. Run the checker independently of the agent's own test command. 4. Record attempts, wall-clock time and tokens per success alongside the binary result. 5. Add partial-credit assertions for intermediate state and report them separately. 6. Audit the checker with a deliberately incomplete solution to confirm it fails.
- Write the end-state assertion first: which row, file content or API response proves completion.
- Reset the environment to a known state before each attempt and record the reset hash.
- Run the checker independently of the agent's own test command.
- Record attempts, wall-clock time and tokens per success alongside the binary result.
- Add partial-credit assertions for intermediate state and report them separately.
- Audit the checker with a deliberately incomplete solution to confirm it fails.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Binary end-state check | Unambiguous | No partial signal | Task has several valid end states |
| Test-suite check | Objective | Tests may be incomplete | Behaviour is visual or interactive |
| Human judge of success | Handles open goals | Cost and disagreement | Scale matters |
| Partial-credit rubric | Rich signal | Gameable, harder to compare | You need a clean headline number |
Good for
Unambiguous
Cost
No partial signal
Fails when
Task has several valid end states
- Task has several valid end states
- Behaviour is visual or interactive
- Scale matters
- You need a clean headline number
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Binary end-state check is good for Unambiguous, costs No partial signal, and fails when Task has several valid end states. Test-suite check is good for Objective, costs Tests may be incomplete, and fails when Behaviour is visual or interactive. Human judge of success is good for Handles open goals, costs Cost and disagreement, and fails when Scale matters. Partial-credit rubric is good for Rich signal, costs Gameable, harder to compare, and fails when You need a clean headline number
Failure modes
- Self-reported success. The agent claims completion and the checker is its own summary. Debug: run the external assertion and compare it with the claim.
- State leakage. A previous run left the environment half-complete and success becomes trivial. Debug: diff the environment against the reset snapshot before each attempt.
- Checker blind spot. A stub implementation passes because the test only checks the return type. Debug: run the checker against a known-bad patch and confirm it fails.
In production
Track success rate with cost and attempts per task, and audit the checker on a schedule. Alert when success rises without a change in task difficulty.
Common misconceptions
People think: An agent that says it finished has finished.
Actually: Completion claims are unverified text, and only the environment state is evidence.
People think: A higher success rate means a better agent.
Actually: It depends on the attempt budget, the checker and the task mix.
Interview angle
Define the end state, the reset and the independent checker, then report cost and attempts with the rate. A strong answer describes how the checker is audited. Accepting the agent's completion message, with no environment reset, is the weak version.
Sources
- Yao et al., tau-bench — tool-agent-user tasks scored against a final database state.
- Liu et al., AgentBench — multi-environment evaluation of agent task completion.
- Zhou et al., WebArena — realistic web tasks with functional correctness checks.