Skip to content
AI Engineer Guide.

concept · evaluation · intermediate · concept 18 of 28

Task success

0 of 28 complete

Why it matters

An agent works across many steps, and the only defensible outcome measure is whether the environment reached the required state. That state has to be asserted independently, because the agent's own report is text and text is exactly what the model produces well. Task success is the anchor metric for agents, and its credibility depends entirely on the checker behind it.

Mental model

It is a delivery confirmed by signature, not by the courier's message. The metaphor breaks because some goals have no signature, only a reasonable stopping point that someone has to define before the work starts, and that definition becomes part of the metric.

How it works

Mechanism, step by stepInteractive
  1. 1Write the end-state assertion first: which row, file content or API response proves completion.
  2. 2Reset the environment to a known state before each attempt and record the reset hash.
  3. 3Run the checker independently of the agent's own test command.
  4. 4Record attempts, wall-clock time and tokens per success alongside the binary result.
  5. 5Add partial-credit assertions for intermediate state and report them separately.
  6. 6Audit the checker with a deliberately incomplete solution to confirm it fails.

Step 1 of 6

Write the end-state assertion first: which row, file content or API response proves completion.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Write the end-state assertion first: which row, file content or API response proves compl… 2. Reset the environment to a known state before each attempt and record the reset hash. 3. Run the checker independently of the agent's own test command. 4. Record attempts, wall-clock time and tokens per success alongside the binary result. 5. Add partial-credit assertions for intermediate state and report them separately. 6. Audit the checker with a deliberately incomplete solution to confirm it fails.

  1. Write the end-state assertion first: which row, file content or API response proves completion.
  2. Reset the environment to a known state before each attempt and record the reset hash.
  3. Run the checker independently of the agent's own test command.
  4. Record attempts, wall-clock time and tokens per success alongside the binary result.
  5. Add partial-credit assertions for intermediate state and report them separately.
  6. Audit the checker with a deliberately incomplete solution to confirm it fails.

Trade-offs

ChoiceGood forCostFails when
Binary end-state checkUnambiguousNo partial signalTask has several valid end states
Test-suite checkObjectiveTests may be incompleteBehaviour is visual or interactive
Human judge of successHandles open goalsCost and disagreementScale matters
Partial-credit rubricRich signalGameable, harder to compareYou need a clean headline number
Trade-off mapInteractive

Good for

Unambiguous

Cost

No partial signal

Fails when

Task has several valid end states

  1. Task has several valid end states
  2. Behaviour is visual or interactive
  3. Scale matters
  4. You need a clean headline number

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Binary end-state check is good for Unambiguous, costs No partial signal, and fails when Task has several valid end states. Test-suite check is good for Objective, costs Tests may be incomplete, and fails when Behaviour is visual or interactive. Human judge of success is good for Handles open goals, costs Cost and disagreement, and fails when Scale matters. Partial-credit rubric is good for Rich signal, costs Gameable, harder to compare, and fails when You need a clean headline number

Failure modes

  • Self-reported success. The agent claims completion and the checker is its own summary. Debug: run the external assertion and compare it with the claim.
  • State leakage. A previous run left the environment half-complete and success becomes trivial. Debug: diff the environment against the reset snapshot before each attempt.
  • Checker blind spot. A stub implementation passes because the test only checks the return type. Debug: run the checker against a known-bad patch and confirm it fails.

In production

Track success rate with cost and attempts per task, and audit the checker on a schedule. Alert when success rises without a change in task difficulty.

Common misconceptions

People think: An agent that says it finished has finished.

Actually: Completion claims are unverified text, and only the environment state is evidence.

People think: A higher success rate means a better agent.

Actually: It depends on the attempt budget, the checker and the task mix.

Interview angle

Define the end state, the reset and the independent checker, then report cost and attempts with the rate. A strong answer describes how the checker is audited. Accepting the agent's completion message, with no environment reset, is the weak version.

Sources