Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 31 of 37

Traces

0 of 37 complete

Why it matters

When an agent gives a bad answer the cause is usually several steps back: a tool returned a stale record, an observation was truncated, a retry duplicated a call. Aggregate metrics cannot locate that. A trace reconstructs the path with enough fidelity that the failing step is identifiable and the run can be re-executed against a fix.

Mental model

A trace is a flight data recorder for one run, with every instrument reading on one timeline. The metaphor breaks because a recorder is passive and complete, while a trace is sampled and shaped by what you chose to log, so the missing field is often the one you need.

How it works

One run, two gaps in the recordInteractive
workretrygap · no span
0.3 s

Event · step 1 of 6

0.3 s · the plan call returns

At 0.3 s

The run span opens at upload and the first model call is a child of it. Plan runs 0.02 to 0.28 s and returns a tool choice; one span has completed by the playhead.

On screen at this event

  • run
  • plan

Next at 0.7 s: 0.7 s · retrieval joins the trace

Spans completed
1
Tool calls
0
Unrecorded
0 ms
Tool output
—
Step 1/6 · 0.3 s · the plan call returns

Figure. A synthetic 3.5 s agent run on one clock: plan, retrieval, a tool call whose observation was truncated, a 180 ms window with no span at all, then a retry of the same lookup. Timings and field widths are illustrative.

What to notice. The 180 ms gap is an uninstrumented code path, not idle time, and the record lane draws it as a hole. The retry is a second lookup for the same key, so one run pays twice and the trace shows the duplicate. A trace that cannot show the tool output cannot explain the answer that used it: record the output or a reference and size, and leave no uninstrumented branch. Sampling and retention are not shown.

Text version

Five lanes over 3.5 seconds: run, model call, retrieval, tool call and record. The run span is 0 to 3.48 s. Plan runs 0.02 to 0.28 s, search 0.32 to 0.66 s, lookup 0.72 to 1.18 s with its output truncated. The record lane is solid 0 to 1.2 s, a red hatched gap 1.2 to 1.38 s labelled no span, then solid to 3.48 s. A re-ask model call runs 1.4 to 1.76 s and a dashed retry lookup 1.78 to 2.16 s. The answer model call runs 2.22 to 3.2 s and a reply 3.2 to 3.3 s. Counters: spans completed 1, 2, 3, 3, 4, 8; tool calls 0, 0, 1, 1, 1, 2; unrecorded 0, 0, 0, 180, 180, 180 ms; tool output truncated from step 3.

  1. Define the span hierarchy (run, step, model call, tool call, retrieval) under one trace id.
  2. Record per span the input, output, model and prompt version, token counts, latency and status.
  3. Attach ids from external systems such as ticket, order or document so the trace joins to the real world.
  4. Redact sensitive fields at write time and set retention per field rather than per store.
  5. Sample successes deterministically and capture errors in full.
  6. Expose a replay path that re-runs a stored trajectory against a new prompt or model.

Trade-offs

ChoiceGood forCostFails when
Full captureBest debugging fidelityStorage cost and privacy exposurePrompts carry regulated data
Sampled captureAffordable at volumeRare failures may be missedFailure rate is low and impact is high
Log lines onlyCheap and familiarNo parent-child structureRuns involve nested tool calls
Trace with replayTurns incidents into test casesReplay harness and version pinningTools have side effects that cannot be re-run
Trade-off mapInteractive

Good for

Best debugging fidelity

Cost

Storage cost and privacy exposure

Fails when

Prompts carry regulated data

  1. Prompts carry regulated data
  2. Failure rate is low and impact is high
  3. Runs involve nested tool calls
  4. Tools have side effects that cannot be re-run

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Full capture is good for Best debugging fidelity, costs Storage cost and privacy exposure, and fails when Prompts carry regulated data. Sampled capture is good for Affordable at volume, costs Rare failures may be missed, and fails when Failure rate is low and impact is high. Log lines only is good for Cheap and familiar, costs No parent-child structure, and fails when Runs involve nested tool calls. Trace with replay is good for Turns incidents into test cases, costs Replay harness and version pinning, and fails when Tools have side effects that cannot be re-run

Failure modes

  • Broken trace id. Spans from one run cannot be assembled into a tree. Debug: query a known run and assert every span shares one trace id and a valid parent.
  • Missing tool output. The trace shows a call but not what came back, so the answer cannot be explained. Debug: assert every tool span has an output or an explicit error field.
  • Unbounded retention. Trace storage grows without a deletion policy and becomes a liability. Debug: sample the oldest partitions and confirm expiry matches the declared retention.

In production

Track trace coverage per run, storage per thousand runs and the share of runs that can be replayed. Alert when coverage drops after a deploy, which usually means a new code path is uninstrumented.

Common misconceptions

People think: Application logs are enough to debug an agent.

Actually: Logs lack the parent-child relation and the exact prompt, so reconstructing a run is guesswork.

People think: Traces are only for performance.

Actually: For agents the trace is the primary correctness artefact, because the path determines the answer.

Interview angle

Strong answers define the span model, the join keys and the redaction policy, then explain how a trace becomes a regression case. Weak answers name an observability vendor and never describe what one span contains.

Sources