Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 21 of 38

LLM tracing

0 of 38 complete

Why it matters

A model call is opaque from the outside: the prompt, the context, the sampled output and the timing all live inside one request. When an answer is wrong, the useful question is which input differed from the working case, and that requires the retrieved chunks and the exact rendered prompt, not just the final text. Tracing preserves that evidence cheaply enough to keep it for every request you might need to explain.

Mental model

A trace is a flight recorder for one conversation: every instrument reading along the way, kept together. The analogy breaks because traces are also evaluation data, so retention and access rules matter more than for a black box that is never read.

How it works

A waterfall with a dropped tool spanInteractive
workgap · no spanevent
710 ms

Event · step 1 of 5

Root through prompt build

At 710 ms

The request span is the root and carries the trace id every child needs. Search ran 90–430, rerank 430–640 with eight candidates, and prompt build 640–710 recorded template v8 and its token count. All four are on one clock.

On screen at this event

  • request
  • search
  • rerank
  • prompt

Next at 980 ms: TTFT is a child, streaming another

Spans expected
4
Spans exported
4
Largest attribute
—
Complete
100%
Step 1/5 · Root through prompt build

Figure. A synthetic request on a 2.5 s clock: retrieval, rerank, prompt build, model call with TTFT and streaming, then the tool call whose span the exporter dropped because its 1.2 MB attribute exceeded the exporter limit. The 1.2 MB size is illustrative.

What to notice. The hole at 2,140 ms is not idle time: the tool ran and returned a document, but the span carrying it was discarded. The trace still looks successful at the root, which is the dangerous part: the answer can be explained everywhere except the call that supplied the evidence. Move bodies to object storage and keep a reference and size under the exporter limit. The 1.2 MB attribute is illustrative; sampling is not shown and this trace is retained.

Text version

Six lanes: request, retrieval, rerank, prompt, model and tool. The root request span runs 0–2,450 ms. Search runs 90–430, rerank 430–640, prompt build 640–710. The model span 710–2,140 has two children on a second row: TTFT 710–980 and streaming 980–2,140. The tool lane has a dashed gap 2,140–2,380 labelled dropped span · 1.2 MB, with a red marker at 2,140 reading exporter limit. The response runs 2,380–2,450. Counters: expected 4, 7, 8, 9, 9; exported 4, 7, 7, 8, 8; largest attribute 1.2 MB; completeness 100, 100, 88, 89, 89 percent.

  1. Define the span tree before choosing a backend: request, retrieval, rerank, prompt build, model call, tool call, response.
  2. Propagate one trace id from the edge through every internal call, including asynchronous workers that continue after the response.
  3. Record the attributes you would need in an incident: model revision, template version, token counts, finish reason, retrieval document ids.
  4. Decide body capture per environment: full in staging, redacted and sampled in production, full on error.
  5. Set retention per field, with a deletion path that can be tested.
  6. Link traces to evaluation runs so a failing sample can be replayed against a candidate prompt.

Trade-offs

ChoiceGood forCostFails when
Full body captureBest debugging and replayStorage cost and privacy exposurePrompts contain regulated data
Metadata onlyCheap and low riskCannot reconstruct the prompt that failedThe bug is in the rendered context
In-process instrumentationRich application contextVendor coupling and upgrade workYou need to change backends
Gateway instrumentationLanguage-agnostic and centralMisses application-level stepsThe failure is in retrieval or prompt assembly
Trade-off mapInteractive

Good for

Best debugging and replay

Cost

Storage cost and privacy exposure

Fails when

Prompts contain regulated data

  1. Prompts contain regulated data
  2. The bug is in the rendered context
  3. You need to change backends
  4. The failure is in retrieval or prompt assembly

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Full body capture is good for Best debugging and replay, costs Storage cost and privacy exposure, and fails when Prompts contain regulated data. Metadata only is good for Cheap and low risk, costs Cannot reconstruct the prompt that failed, and fails when The bug is in the rendered context. In-process instrumentation is good for Rich application context, costs Vendor coupling and upgrade work, and fails when You need to change backends. Gateway instrumentation is good for Language-agnostic and central, costs Misses application-level steps, and fails when The failure is in retrieval or prompt assembly

Failure modes

  • Broken trace across async work. A background worker starts a new trace and the job cannot be linked to the request that queued it. Debug: assert the incoming trace id appears on the worker's root span.
  • Unsampled success traffic. Only errors are retained and a slow quality decline has no baseline. Debug: sample successes deterministically by request id and confirm the rate matches the configured fraction.
  • Oversized spans. Full documents in attributes exceed exporter limits and spans are dropped. Debug: check span size against the exporter limit and move large payloads to object storage with a reference.

In production

Alert on export failures and on a drop in spans per request, because a silent sampling change looks exactly like a traffic change. Keep one retention policy per data class and verify deletion with a scheduled test.

Common misconceptions

People think: Tracing is logging with a nicer interface.

Actually: A trace is a causal tree with timing and joins; flat logs cannot tell you which retrieval call fed which generation.

People think: You need every prompt to debug production.

Actually: A redacted sample plus full capture on errors answers most questions with far less exposure.

Interview angle

Answer with the span tree, the join key and the capture policy, then give the incident question the design serves. A strong answer separates metadata from bodies and explains sampling. Weak answers name a vendor and assume traces are free to store.

Sources