Why it matters
Outcome-only scoring hides the difference between an agent that solved a task and one that got there by deleting and recreating data. For systems that act, the path is the product: a run that passes by an unsafe route will fail differently next time. Trajectory evaluation makes the path comparable, so a change that shortens the route can be told apart from one that got lucky.
Mental model
Judging a trajectory is reviewing a chess game, not only the result on the board. The metaphor breaks because chess has one legal move set and a known objective, while agent tasks admit many valid routes, so the reviewer judges legitimacy rather than similarity.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Modelled score distributions for the same runs, measured twice: once on the outcome and once on the path. 84% of runs are clean and score highly on both; 16% reach a high outcome by a destructive path and sit near 34 on path. The two curves are marginals, not a scatter, and the scores are illustrative.
What to notice. At an 80-point bar, 8.8% of runs fall below it on outcome and 23.7% on path. The gap is the 16% destructive tail: those runs pass the outcome check at a median of 84 while their path median is 34. An outcome-only screen calls them successes; the path score is the only view that separates them.
Text version
Two modelled score distributions from 0 to 100. The outcome curve is high and narrow, peaking near 89 at 34% of runs per five points, with an 8.8% share below an 80-point bar. The path curve peaks near 88 but carries a second bump at 34 reaching 3.5% per five points; 23.7% of runs fall below the 80-point bar on path. A vertical ship-bar line moves from 50 to 95 with a slider, and a panel reports the share of runs below it on each score plus the 16% destructive tail.
- Define the properties a good path must have: allowed tools, forbidden tools, what must be verified before a commit.
- Capture full trajectories with arguments and observations, not just the final message.
- Score with deterministic checks first and a model judge only for the residue.
- Measure efficiency separately: steps, tokens, wall-clock and duplicate calls.
- Compare candidate and baseline on the same tasks, reporting outcome and path scores together.
- Promote failing trajectories into the regression suite with their expected properties.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Outcome only | Objective and cheap | Rewards unsafe or lucky paths | Actions have side effects |
| Exact path match | Trivial to compute | Rejects legitimate alternatives | Many valid routes exist |
| Property checks | Catches unsafe paths | Needs a property list per task class | Properties are hard to state |
| Model judge on trajectories | Handles nuance | Cost, bias and non-determinism | The judge cannot see tool effects |
Good for
Objective and cheap
Cost
Rewards unsafe or lucky paths
Fails when
Actions have side effects
- Actions have side effects
- Many valid routes exist
- Properties are hard to state
- The judge cannot see tool effects
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Outcome only is good for Objective and cheap, costs Rewards unsafe or lucky paths, and fails when Actions have side effects. Exact path match is good for Trivial to compute, costs Rejects legitimate alternatives, and fails when Many valid routes exist. Property checks is good for Catches unsafe paths, costs Needs a property list per task class, and fails when Properties are hard to state. Model judge on trajectories is good for Handles nuance, costs Cost, bias and non-determinism, and fails when The judge cannot see tool effects
Failure modes
- Path-blind scoring. A destructive run scores as a pass because the end state matched. Debug: add a check for forbidden tools and confirm it fires on a seeded bad trajectory.
- Verbosity bias. Longer trajectories score higher because they look thorough. Debug: correlate judge scores with step count and inspect the top-scoring runs.
- Uncaptured steps. The trajectory lacks tool arguments and cannot be scored. Debug: assert each step has an argument payload before evaluation runs.
In production
Report outcome and path scores together for every candidate, and track the distribution of steps per task. Keep a fixed set of bad trajectories as negative tests for the scorer itself.
Common misconceptions
People think: A correct answer means a good trajectory.
Actually: Correctness can be reached by routes you must never ship, and the next run may not be lucky.
People think: Trajectory evaluation requires a model judge.
Actually: Most properties are deterministic checks over recorded steps; judges handle the residue.
Interview angle
Strong answers define path properties, separate deterministic checks from judged ones, and report efficiency alongside success. Weak answers say the final answer is graded, which cannot distinguish a safe run from a lucky one.
Sources
- Liu et al., AgentBench: Evaluating LLMs as Agents — multi-environment evaluation of agent behaviour beyond final answers.
- Zhou et al., WebArena: A Realistic Web Environment for Building Autonomous Agents — functional checks over the resulting state of a multi-step run.
- Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — biases and limits of model graders.