Why it matters
An agent is a program whose control flow is chosen at runtime, so unit tests that assert on function outputs miss the thing that broke. Its output is a sequence of side effects in an environment, and the same instruction can succeed through several paths. Evaluation needs tasks with verifiable end states, a harness that resets the world between trials, and metrics that separate capability failures from plumbing failures.
Mental model
Evaluating an agent is auditioning a chef by watching a whole service, not tasting one plate. The metaphor breaks because a kitchen's pass or fail is externally defined, while here you must write the checker, and the checker can be wrong in the same way the agent is.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Modelled distribution of a 40-task suite's pass rate across reruns, with one, four and sixteen trials per task. The suite's true rate is 68%; the curves are normal approximations to a discrete score (2.5-point steps at one trial). At the 30-of-40 gate, 18.2% of single runs clear it and 1.7% do at four trials. The binomial model is stated in the footnote and every point is generated, not measured.
What to notice. One trial per task gives the suite score a spread of ±6.3 points, and 18.2% of single runs reach 30 of 40 (the 75% gate) even though the true rate is 68%. Four trials halve the spread and 1.7% reach the gate; sixteen cut it to ±1.6 points, where the gate is almost never cleared by noise. A single report cannot tell a 68% agent from a 75% one; the trial count is part of the number.
Text version
Three modelled bell curves of suite pass rate from 40 to 96%. The widest, one trial per task, is centred on 68% with a standard deviation of 6.3 points and peaks at 12.6% of reruns per two-point bin. Four trials peaks at 25.2% and sixteen trials at 50.5%. A vertical warning line marks the 75% gate, which is 30 of 40 tasks; 18.2% of single runs reach 30 or more, and 1.7% do at four trials. The smooth curve leaves 13.4% past the line because the gate falls on a task boundary, so the panel prints the discrete tail. A baseline note marks the true 68% rate. Three steps reveal one run, four runs and sixteen runs; the resting frame shows all three.
- Write each task as an initial state, an instruction and a success predicate that code can check.
- Reset the database, filesystem and mock APIs between trials, and seed randomness.
- Run each task several times and report a pass rate with the trial count, never a single result.
- Record the trajectory: tool calls, arguments, observations, tokens and wall-clock time.
- Report outcome, process and efficiency separately so a plumbing failure is not read as a capability failure.
- Keep a frozen regression suite and a rotating fresh suite, so the agent cannot overfit the fixed tasks.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| End-state checks only | Objective and cheap to grade | Rewards lucky or destructive paths | Side effects are irreversible or partially correct |
| Trajectory checks | Catches bad process early | Brittle to legitimate alternative routes | Many valid paths exist |
| Simulated environment | Repeatable, fast, safe | Misses real API and UI behaviour | The product depends on live third-party systems |
| Live environment | Highest fidelity | Costly, flaky, hard to reset | Tasks mutate shared data |
Good for
Objective and cheap to grade
Cost
Rewards lucky or destructive paths
Fails when
Side effects are irreversible or partially correct
- Side effects are irreversible or partially correct
- Many valid paths exist
- The product depends on live third-party systems
- Tasks mutate shared data
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: End-state checks only is good for Objective and cheap to grade, costs Rewards lucky or destructive paths, and fails when Side effects are irreversible or partially correct. Trajectory checks is good for Catches bad process early, costs Brittle to legitimate alternative routes, and fails when Many valid paths exist. Simulated environment is good for Repeatable, fast, safe, costs Misses real API and UI behaviour, and fails when The product depends on live third-party systems. Live environment is good for Highest fidelity, costs Costly, flaky, hard to reset, and fails when Tasks mutate shared data
Failure modes
- Leaky environment. Later trials inherit state from earlier ones and pass rates drift. Debug: run the suite twice from a clean snapshot and compare the two results.
- Permissive grader. A loose success predicate scores broken runs as passes. Debug: seed known-failing trajectories and confirm the grader rejects them.
- Single-run verdict. A flaky task flips the result on rerun. Debug: repeat one task ten times and inspect the spread before trusting any delta.
In production
Run the suite on every prompt or model change and track pass rate, median steps and cost per task over time. Keep a smoke suite fast enough for pull requests and a larger one for release candidates.
Common misconceptions
People think: A higher task success rate means a better agent.
Actually: Success without a step and cost budget hides a system that will not survive real traffic.
People think: Public agent benchmarks substitute for your own suite.
Actually: They measure different tools, data and tolerances; they rank research, not your product.
Interview angle
A strong answer defines the task distribution, the success predicate and the reset discipline before naming a metric, then separates capability from reliability and reports variance across repeats. Weak answers quote one benchmark score and never explain how the environment was made reproducible.
Sources
- Yao et al., τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — task success with a simulated user and a database end-state check.
- Zhou et al., WebArena: A Realistic Web Environment for Building Autonomous Agents — reproducible environments and functional correctness checks for agent tasks.
- Breck et al., The ML Test Score — a rubric for testing data and model-dependent systems.