Why it matters
A model call is opaque from the outside: the prompt, the context, the sampled output and the timing all live inside one request. When an answer is wrong, the useful question is which input differed from the working case, and that requires the retrieved chunks and the exact rendered prompt, not just the final text. Tracing preserves that evidence cheaply enough to keep it for every request you might need to explain.
Mental model
A trace is a flight recorder for one conversation: every instrument reading along the way, kept together. The analogy breaks because traces are also evaluation data, so retention and access rules matter more than for a black box that is never read.
How it works
Event · step 1 of 5
Root through prompt build
At 710 ms
The request span is the root and carries the trace id every child needs. Search ran 90–430, rerank 430–640 with eight candidates, and prompt build 640–710 recorded template v8 and its token count. All four are on one clock.
On screen at this event
- request
- search
- rerank
- prompt
Next at 980 ms: TTFT is a child, streaming another
- Spans expected
- 4
- Spans exported
- 4
- Largest attribute
- —
- Complete
- 100%
Figure. A synthetic request on a 2.5 s clock: retrieval, rerank, prompt build, model call with TTFT and streaming, then the tool call whose span the exporter dropped because its 1.2 MB attribute exceeded the exporter limit. The 1.2 MB size is illustrative.
What to notice. The hole at 2,140 ms is not idle time: the tool ran and returned a document, but the span carrying it was discarded. The trace still looks successful at the root, which is the dangerous part: the answer can be explained everywhere except the call that supplied the evidence. Move bodies to object storage and keep a reference and size under the exporter limit. The 1.2 MB attribute is illustrative; sampling is not shown and this trace is retained.
Text version
Six lanes: request, retrieval, rerank, prompt, model and tool. The root request span runs 0–2,450 ms. Search runs 90–430, rerank 430–640, prompt build 640–710. The model span 710–2,140 has two children on a second row: TTFT 710–980 and streaming 980–2,140. The tool lane has a dashed gap 2,140–2,380 labelled dropped span · 1.2 MB, with a red marker at 2,140 reading exporter limit. The response runs 2,380–2,450. Counters: expected 4, 7, 8, 9, 9; exported 4, 7, 7, 8, 8; largest attribute 1.2 MB; completeness 100, 100, 88, 89, 89 percent.
- Define the span tree before choosing a backend: request, retrieval, rerank, prompt build, model call, tool call, response.
- Propagate one trace id from the edge through every internal call, including asynchronous workers that continue after the response.
- Record the attributes you would need in an incident: model revision, template version, token counts, finish reason, retrieval document ids.
- Decide body capture per environment: full in staging, redacted and sampled in production, full on error.
- Set retention per field, with a deletion path that can be tested.
- Link traces to evaluation runs so a failing sample can be replayed against a candidate prompt.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Full body capture | Best debugging and replay | Storage cost and privacy exposure | Prompts contain regulated data |
| Metadata only | Cheap and low risk | Cannot reconstruct the prompt that failed | The bug is in the rendered context |
| In-process instrumentation | Rich application context | Vendor coupling and upgrade work | You need to change backends |
| Gateway instrumentation | Language-agnostic and central | Misses application-level steps | The failure is in retrieval or prompt assembly |
Good for
Best debugging and replay
Cost
Storage cost and privacy exposure
Fails when
Prompts contain regulated data
- Prompts contain regulated data
- The bug is in the rendered context
- You need to change backends
- The failure is in retrieval or prompt assembly
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Full body capture is good for Best debugging and replay, costs Storage cost and privacy exposure, and fails when Prompts contain regulated data. Metadata only is good for Cheap and low risk, costs Cannot reconstruct the prompt that failed, and fails when The bug is in the rendered context. In-process instrumentation is good for Rich application context, costs Vendor coupling and upgrade work, and fails when You need to change backends. Gateway instrumentation is good for Language-agnostic and central, costs Misses application-level steps, and fails when The failure is in retrieval or prompt assembly
Failure modes
- Broken trace across async work. A background worker starts a new trace and the job cannot be linked to the request that queued it. Debug: assert the incoming trace id appears on the worker's root span.
- Unsampled success traffic. Only errors are retained and a slow quality decline has no baseline. Debug: sample successes deterministically by request id and confirm the rate matches the configured fraction.
- Oversized spans. Full documents in attributes exceed exporter limits and spans are dropped. Debug: check span size against the exporter limit and move large payloads to object storage with a reference.
In production
Alert on export failures and on a drop in spans per request, because a silent sampling change looks exactly like a traffic change. Keep one retention policy per data class and verify deletion with a scheduled test.
Common misconceptions
People think: Tracing is logging with a nicer interface.
Actually: A trace is a causal tree with timing and joins; flat logs cannot tell you which retrieval call fed which generation.
People think: You need every prompt to debug production.
Actually: A redacted sample plus full capture on errors answers most questions with far less exposure.
Interview angle
Answer with the span tree, the join key and the capture policy, then give the incident question the design serves. A strong answer separates metadata from bodies and explains sampling. Weak answers name a vendor and assume traces are free to store.
Sources
- OpenTelemetry, Traces — span structure, context propagation and sampling.
- W3C, Trace Context — the header format that carries trace identity across services.
- OpenTelemetry GenAI semantic conventions — attribute names for model spans.