Skip to content
AI Engineer Guide.

concept · evaluation · intermediate · concept 19 of 28

Trajectory evaluation

0 of 28 complete

Why it matters

Two runs can reach the same end state, one by reading the right file and running one test, the other by deleting a cache, retrying a broken command and guessing. Outcome-only evaluation treats them as equal. Trajectory evaluation looks at the steps to find unsafe shortcuts, loops and actions that were not supported by evidence, which is what predicts whether the next run will succeed.

Mental model

It is judging a climb by the route as well as the summit. The metaphor breaks because many routes reach the top and only some are reckless, which the summit photo cannot show, so the route has to be recorded while it happens.

How it works

Mechanism, step by stepInteractive
  1. 1Record every step with tool name, arguments, a result summary and token cost.
  2. 2Define allowed and forbidden step classes, such as no writes outside the workspace.
  3. 3Score necessity by checking whether removing a step still reaches the outcome.
  4. 4Score grounding by checking whether each action cites evidence the agent had at that moment.
  5. 5Compare outcome and trajectory scores and inspect cases where they diverge.
  6. 6Report step-count and cost distributions rather than only the pass rate.

Step 1 of 6

Record every step with tool name, arguments, a result summary and token cost.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Record every step with tool name, arguments, a result summary and token cost. 2. Define allowed and forbidden step classes, such as no writes outside the workspace. 3. Score necessity by checking whether removing a step still reaches the outcome. 4. Score grounding by checking whether each action cites evidence the agent had at that mome… 5. Compare outcome and trajectory scores and inspect cases where they diverge. 6. Report step-count and cost distributions rather than only the pass rate.

  1. Record every step with tool name, arguments, a result summary and token cost.
  2. Define allowed and forbidden step classes, such as no writes outside the workspace.
  3. Score necessity by checking whether removing a step still reaches the outcome.
  4. Score grounding by checking whether each action cites evidence the agent had at that moment.
  5. Compare outcome and trajectory scores and inspect cases where they diverge.
  6. Report step-count and cost distributions rather than only the pass rate.

Trade-offs

ChoiceGood forCostFails when
Constraint checkingObjective and cheapOnly catches known bad patternsA novel failure appears
Judge-scored trajectoryNuancedCost and judge biasVolume is high
Exact path matchingSimple to computeRejects valid alternativesTask has many solutions
Outcome onlyCheap and clearHides unsafe or wasteful pathsSafety matters
Trade-off mapInteractive

Good for

Objective and cheap

Cost

Only catches known bad patterns

Fails when

A novel failure appears

  1. A novel failure appears
  2. Volume is high
  3. Task has many solutions
  4. Safety matters

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Constraint checking is good for Objective and cheap, costs Only catches known bad patterns, and fails when A novel failure appears. Judge-scored trajectory is good for Nuanced, costs Cost and judge bias, and fails when Volume is high. Exact path matching is good for Simple to compute, costs Rejects valid alternatives, and fails when Task has many solutions. Outcome only is good for Cheap and clear, costs Hides unsafe or wasteful paths, and fails when Safety matters

Failure modes

  • Lucky success. The agent reaches the answer through a forbidden shortcut and outcome scoring passes it. Debug: check the forbidden-step log for successful runs.
  • Loop masking. The agent retries the same failing command until a timeout and then succeeds by accident. Debug: count repeated identical tool calls per run.
  • Judge path preference. The scorer rewards the style it expects and penalises a shorter valid route. Debug: compare scores for two valid paths of different lengths.

In production

Track steps per success, forbidden-step hits and repeated-call counts. Sample trajectories for human review each release.

Common misconceptions

People think: Only the final state matters.

Actually: It hides unsafe shortcuts and non-reproducible luck.

People think: There is one correct trajectory.

Actually: There are usually many, so evaluate constraints and evidence rather than the exact sequence.

Interview angle

Score outcome and process together, define forbidden step classes and check grounding per step. A strong answer covers loops and lucky successes. Comparing only final states, with no view of the route, is the weak response.

Sources