Why it matters
Two runs can reach the same end state, one by reading the right file and running one test, the other by deleting a cache, retrying a broken command and guessing. Outcome-only evaluation treats them as equal. Trajectory evaluation looks at the steps to find unsafe shortcuts, loops and actions that were not supported by evidence, which is what predicts whether the next run will succeed.
Mental model
It is judging a climb by the route as well as the summit. The metaphor breaks because many routes reach the top and only some are reckless, which the summit photo cannot show, so the route has to be recorded while it happens.
How it works
- 1Record every step with tool name, arguments, a result summary and token cost.
- 2Define allowed and forbidden step classes, such as no writes outside the workspace.
- 3Score necessity by checking whether removing a step still reaches the outcome.
- 4Score grounding by checking whether each action cites evidence the agent had at that moment.
- 5Compare outcome and trajectory scores and inspect cases where they diverge.
- 6Report step-count and cost distributions rather than only the pass rate.
Step 1 of 6
Record every step with tool name, arguments, a result summary and token cost.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Record every step with tool name, arguments, a result summary and token cost. 2. Define allowed and forbidden step classes, such as no writes outside the workspace. 3. Score necessity by checking whether removing a step still reaches the outcome. 4. Score grounding by checking whether each action cites evidence the agent had at that mome… 5. Compare outcome and trajectory scores and inspect cases where they diverge. 6. Report step-count and cost distributions rather than only the pass rate.
- Record every step with tool name, arguments, a result summary and token cost.
- Define allowed and forbidden step classes, such as no writes outside the workspace.
- Score necessity by checking whether removing a step still reaches the outcome.
- Score grounding by checking whether each action cites evidence the agent had at that moment.
- Compare outcome and trajectory scores and inspect cases where they diverge.
- Report step-count and cost distributions rather than only the pass rate.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Constraint checking | Objective and cheap | Only catches known bad patterns | A novel failure appears |
| Judge-scored trajectory | Nuanced | Cost and judge bias | Volume is high |
| Exact path matching | Simple to compute | Rejects valid alternatives | Task has many solutions |
| Outcome only | Cheap and clear | Hides unsafe or wasteful paths | Safety matters |
Good for
Objective and cheap
Cost
Only catches known bad patterns
Fails when
A novel failure appears
- A novel failure appears
- Volume is high
- Task has many solutions
- Safety matters
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Constraint checking is good for Objective and cheap, costs Only catches known bad patterns, and fails when A novel failure appears. Judge-scored trajectory is good for Nuanced, costs Cost and judge bias, and fails when Volume is high. Exact path matching is good for Simple to compute, costs Rejects valid alternatives, and fails when Task has many solutions. Outcome only is good for Cheap and clear, costs Hides unsafe or wasteful paths, and fails when Safety matters
Failure modes
- Lucky success. The agent reaches the answer through a forbidden shortcut and outcome scoring passes it. Debug: check the forbidden-step log for successful runs.
- Loop masking. The agent retries the same failing command until a timeout and then succeeds by accident. Debug: count repeated identical tool calls per run.
- Judge path preference. The scorer rewards the style it expects and penalises a shorter valid route. Debug: compare scores for two valid paths of different lengths.
In production
Track steps per success, forbidden-step hits and repeated-call counts. Sample trajectories for human review each release.
Common misconceptions
People think: Only the final state matters.
Actually: It hides unsafe shortcuts and non-reproducible luck.
People think: There is one correct trajectory.
Actually: There are usually many, so evaluate constraints and evidence rather than the exact sequence.
Interview angle
Score outcome and process together, define forbidden step classes and check grounding per step. A strong answer covers loops and lucky successes. Comparing only final states, with no view of the route, is the weak response.
Sources
- Ma et al., AgentBoard — fine-grained progress metrics for agent trajectories.
- Yao et al., tau-bench — task state and interaction rules for tool agents.
- Cemri et al., Why Do Multi-Agent LLM Systems Fail? — trajectory-level failure categories.