Why it matters
Classic debugging starts from a stack trace or a failing assertion. An LLM system produces well-formed 200 responses that are simply wrong, and the cause can sit in any of five places: the retrieval set, the prompt assembly order, the model version, the decoding parameters, or the tool result that was fed back in. None of them appear in an application log line like request completed in 812ms.
Without recorded context, every quality report degenerates into a reproduction attempt: ask the user what they typed, hope the corpus has not changed, hope the provider has not moved the model. With it, a bad answer becomes a specific trace you can replay against a fixed index snapshot and a pinned model version.
Observability is also the input to evaluation. A trace store that captures real queries, retrieved ids and outcomes is the raw material for next month's golden set. The two systems should share identifiers, not run in parallel.
Mental model
A flight recorder plus a departure board. The recorder holds the full sequence of events for one flight, and you read it after something goes wrong. The board aggregates: how many flights are late, by how much, on which route. Neither replaces the other — the board tells you where to look, the recorder tells you what happened.
The analogy breaks on completeness. A flight recorder captures everything and is read rarely. At LLM request volumes you cannot store every prompt at full fidelity forever, so you sample traces and redact content. That means the specific request you need may not have been recorded, which is why errors and slow requests must be sampled at a higher rate than successes.
How it works
Event · step 1 of 6
Ingress and retrieval
At 310 ms
The edge opened trace 4bf at ingress and wrote a W3C traceparent header. Search ran from 60 to 310 ms as a child with the query, filters, k and returned ids; rerank starts now. These spans are what let a review see which chunks fed the answer.
On screen at this event
- request · 4bf
- search
- rerank
Next at 420 ms: Queue wait is recorded
- Spans · 4bf
- 4
- Orphan spans
- 0
- Trace ids
- 1
- Empty parent
- 0
Figure. One synthetic request on a 2.6 s clock. Trace 4bf covers ingress, retrieval, queue and the response; the worker starts trace 9c1 and the model and tool spans are orphaned. The 70 ms between the queue and the worker's new root is the missing parent link, not idle time.
What to notice. Trace 4bf has a hole between the queue and the response, and trace 9c1 contains the generation, so the store can explain latency on both sides and cannot join them; the first orphan span starts at 1,050. The failure is context propagation, not missing instrumentation: the worker built a new root span instead of forwarding traceparent. Sampling and retention are not shown; this is one fully captured request. Count empty parent ids and alert on the ratio; the fix is at the hop, not in a new dashboard.
Text version
Six lanes: edge, retrieval, queue, worker, model and tools. Trace 4bf runs as a request span from 0 to 2,500 ms on the edge. Retrieval searches 60–310 and reranks 310–420. The queue waits 420–980. At 1,050 a marker says new trace; the worker lane shows a dashed gap 980–1,050 labelled no span, then a red span 1,050–2,450 labelled new trace · 9c1. The model call 1,100–1,900 and tool lookup 1,900–2,200 sit under that new trace. The response returns 2,450–2,500. Counters: six spans in 4bf, three orphans, two trace ids and one empty parent from the new root onward.
- Assign a trace id at ingress. Propagate it through the queue, the retrieval service, the model gateway and every tool call using the W3C
traceparentheader. If any hop starts a new trace, you have lost the request. - Emit a span per unit of work. Retrieval (query, filters, k, returned ids and scores), reranking (candidate count, model, latency), each model call (provider, model id, resolved version, prompt hash, token counts, TTFT, finish reason), and each tool call (name, argument hash, status, latency).
- Keep content out of span attributes by default. Spans carry identifiers and sizes; the prompt and completion go to a content store keyed by trace id, encrypted, access-controlled, with a short retention. This keeps the tracing backend cheap and the sensitive data in one place you can delete.
- Record the outcome, not just the mechanics. Thumbs, edits, escalations, retries and citation clicks are the labels that make a trace useful for quality work. Join them on trace id or session id.
- Aggregate into metrics. TTFT p50/p95, error rate by provider and status, cost per request, retrieval empty-result rate, refusal rate, tool error rate. Metrics are cheap to keep for a year; traces are not.
- Sample deliberately. Head sampling at a fixed rate is simple but blind. Tail sampling at the collector can keep every error and every slow trace plus a baseline percentage of the rest.
- Alert on symptoms. Page on SLO burn (error rate, latency, task success proxy), not on a single provider 429. See AI SLOs.
Covers OpenTelemetry GenAI semantic conventions (development status) · last verified 2026-11-17
The GenAI conventions are still evolving and attribute names have changed between releases. Pin the semconv version you emit and translate at the collector rather than rewriting instrumentation every quarter.
from opentelemetry import trace
tracer = trace.get_tracer("support-copilot")
def call_model(client, prompt: str, prompt_hash: str, model_id: str):
with tracer.start_as_current_span("llm.call") as span:
span.set_attribute("gen_ai.provider.name", "example-provider")
span.set_attribute("gen_ai.request.model", model_id)
span.set_attribute("prompt.sha256", prompt_hash)
span.set_attribute("prompt.tokens", count_tokens(prompt))
resp = client.complete(prompt)
span.set_attribute("gen_ai.usage.input_tokens", resp.usage.input_tokens)
span.set_attribute("gen_ai.usage.output_tokens", resp.usage.output_tokens)
span.set_attribute("gen_ai.response.finish_reasons", [resp.finish_reason])
span.set_attribute("llm.ttft_ms", resp.ttft_ms)
span.set_attribute("content.ref", store.put(prompt, resp.text))
return respSampling, cardinality and retention
Three numbers decide whether the system survives.
Trace volume. At 1,000,000 requests per day and roughly 8 KB of span data each, you produce about 8 GB per day. Thirty days is 240 GB before compression. That is affordable for spans and not affordable for full prompt content, which is why content lives elsewhere with a shorter window.
Metric cardinality. Each unique label combination is a separate time series. 50 tenants × 8 routes × 4 model versions = 1,600 series, which is fine. Add user id and you get 200,000 × 8 = 1.6 million series, which will overwhelm a Prometheus-style store and inflate cost long before it helps anyone. Keep user and session identifiers in traces; keep metric labels to bounded dimensions.
Sampling bias. If you keep 10% of traces at the head, you keep roughly 10% of failures too. Tail sampling that retains all errors and all traces above the latency objective costs more storage per failure but is the only way to debug the failures you actually care about.
Trade-offs
| Choice | Buys | Costs | Fails when |
|---|---|---|---|
| Full prompt content in spans | One-system debugging | PII risk, storage cost, cardinality | Any regulated data enters the prompt |
| Hashes and pointers only | Cheap, safe spans | A second lookup to debug | The content store has shorter retention than traces |
| Head sampling | Simple, predictable cost | Blind to rare failures | Failure rate is below the sampling rate |
| Tail sampling | All errors and slow traces | Collector complexity and memory | Traffic is too bursty for the collector buffer |
| Vendor observability platform | Fast setup, built-in evals | Per-span pricing, lock-in | Volume grows faster than budget |
Good for
One-system debugging
Cost
PII risk, storage cost, cardinality
Fails when
Any regulated data enters the prompt
- Any regulated data enters the prompt
- The content store has shorter retention than traces
- Failure rate is below the sampling rate
- Traffic is too bursty for the collector buffer
- Volume grows faster than budget
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Full prompt content in spans is good for One-system debugging, costs PII risk, storage cost, cardinality, and fails when Any regulated data enters the prompt. Hashes and pointers only is good for Cheap, safe spans, costs A second lookup to debug, and fails when The content store has shorter retention than traces. Head sampling is good for Simple, predictable cost, costs Blind to rare failures, and fails when Failure rate is below the sampling rate. Tail sampling is good for All errors and slow traces, costs Collector complexity and memory, and fails when Traffic is too bursty for the collector buffer. Vendor observability platform is good for Fast setup, built-in evals, costs Per-span pricing, lock-in, and fails when Volume grows faster than budget
Failure modes
- Broken context propagation. A background worker or queue consumer starts a new trace, so the request looks like two unrelated events. Debug by counting spans whose parent id is empty and checking that the
traceparentheader is forwarded on every hop. - Unredacted prompts in storage. An email address or access token lands in a traced prompt and the store has broad read access. Debug by sampling stored payloads and running your redaction patterns against them; treat a hit as an incident, not a backlog item.
- Errors missing from sampled traces. The dashboard shows a 4% error rate and the trace store shows almost none. Debug by comparing error counts in metrics with error spans in storage, then fix sampling before drawing conclusions.
- Traces without outcomes. Every request is recorded and none is labelled, so you can describe mechanics but not quality. Debug by measuring the join rate between traces and feedback events; if it is low, the feedback path is broken, not the model.
In production
Separate retention by signal: metrics for a year, traces for two to four weeks, prompt content for days unless a regulator or contract requires longer. Document the retention in the privacy notice; it is a data-processing fact, not an internal preference.
Give on-call a single query that takes a trace id and returns the whole story: prompt, retrieved ids, model version, tokens, tool calls, outcome. If that query takes four tools and ten minutes, nobody runs it at 3 a.m.
Budget observability as a percentage of model spend and revisit quarterly. Tracing is easy to switch on and hard to switch off once a team depends on it; decide the sampling policy before the volume arrives, not after the bill. See LLM tracing and AI dashboards.
Common misconceptions
People think: Logging every prompt and completion is observability.
Actually: That is a content archive. Observability also needs structure: which retrieval set, which model version, which tokens, which outcome, and the ability to aggregate across requests.
People think: Traces and metrics are alternatives.
Actually: Metrics find the affected population cheaply and traces explain one member of it. Systems that have only one of the two spend their incidents guessing.
Interview angle
Expect "a customer says the answer was wrong at 3 p.m. yesterday — walk me through finding out why." A strong answer asks for a trace id or a session identifier, then reconstructs query, retrieved chunks, prompt hash, model version and finish reason, and replays the request against the recorded index snapshot. Then it names what is missing if the content retention was too short, and what it would change. See LLM tracing and the production interview category.
Sources
- OpenTelemetry: Traces — spans, context propagation and sampling concepts.
- OpenTelemetry GenAI semantic conventions — attributes for model, token usage and provider.
- Sigelman et al., Dapper, a Large-Scale Distributed Systems Tracing Infrastructure — the design constraints behind low-overhead tracing.
- Google SRE Book, Monitoring Distributed Systems — symptoms versus causes, and why alerting on every cause fails.
Related
- LLM tracing — the LLM-specific span shape.
- OTel GenAI — instrumentation in practice.
- AI dashboards — what to put on the wall.
- AI SLOs — turning signals into promises.
- Retrieval observability — the retrieval half of the trace.