Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 57 of 57

Retrieval observability

0 of 57 complete

Why it matters

An answer is the end of a chain: query rewrite, retrieval, filtering, reranking, assembly and generation. When it is wrong, the only useful question is which link failed, and without a joined record the team guesses. Retrieval observability exists to make that question answerable, which requires capturing intermediate artefacts — candidate ids, scores, admitted passages — not just the final prompt and response.

Mental model

Observability is a flight recorder for each request: inputs, decisions and outputs on one timeline. The metaphor breaks because the recorder also stores the passengers' documents, so privacy and retention are part of the instrument design, and the record is only useful if every instrument writes to the same clock.

How it works

Mechanism, step by stepInteractive
  1. 1Generate one request id at the edge and propagate it through rewrite, retrieval, rerank and generation.
  2. 2Record the original and rewritten query, the filters applied and the retriever configuration for the request.
  3. 3Log candidate ids with scores before and after reranking, and which ones survived assembly.
  4. 4Record the final prompt hash, token counts and the cited passage ids, not the full prompt by default.
  5. 5Define metrics from the traces: zero-result rate, score distribution, rank of the cited chunk and stage latency.
  6. 6Sample successes deterministically and capture full detail on errors and user-flagged answers.

Step 1 of 6

Generate one request id at the edge and propagate it through rewrite, retrieval, rerank and generation.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Generate one request id at the edge and propagate it through rewrite, retrieval, rerank a… 2. Record the original and rewritten query, the filters applied and the retriever configurat… 3. Log candidate ids with scores before and after reranking, and which ones survived assembl… 4. Record the final prompt hash, token counts and the cited passage ids, not the full prompt… 5. Define metrics from the traces: zero-result rate, score distribution, rank of the cited c… 6. Sample successes deterministically and capture full detail on errors and user-flagged ans…

  1. Generate one request id at the edge and propagate it through rewrite, retrieval, rerank and generation.
  2. Record the original and rewritten query, the filters applied and the retriever configuration for the request.
  3. Log candidate ids with scores before and after reranking, and which ones survived assembly.
  4. Record the final prompt hash, token counts and the cited passage ids, not the full prompt by default.
  5. Define metrics from the traces: zero-result rate, score distribution, rank of the cited chunk and stage latency.
  6. Sample successes deterministically and capture full detail on errors and user-flagged answers.

Trade-offs

ChoiceGood forCostFails when
Full prompt captureBest debuggingStorage cost and privacy exposurePrompts contain regulated data
Metadata-only tracesLow risk and cheapCannot reproduce a bad promptThe failure is in wording or ordering
Sampled tracesBounded costRare failures may be missedTraffic is low volume
Synchronous loggingComplete recordsLatency and a failure surface in the request pathThe service is latency-critical
Trade-off mapInteractive

Good for

Best debugging

Cost

Storage cost and privacy exposure

Fails when

Prompts contain regulated data

  1. Prompts contain regulated data
  2. The failure is in wording or ordering
  3. Traffic is low volume
  4. The service is latency-critical

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Full prompt capture is good for Best debugging, costs Storage cost and privacy exposure, and fails when Prompts contain regulated data. Metadata-only traces is good for Low risk and cheap, costs Cannot reproduce a bad prompt, and fails when The failure is in wording or ordering. Sampled traces is good for Bounded cost, costs Rare failures may be missed, and fails when Traffic is low volume. Synchronous logging is good for Complete records, costs Latency and a failure surface in the request path, and fails when The service is latency-critical

Failure modes

  • No join key. Retrieval spans and generation spans cannot be matched, so the failing stage is unknown. Debug: pick one request and verify the same id appears on every span.
  • Only failures traced. Quality drifts on successful requests and nobody sees it. Debug: sample a fixed percentage of successes by request id and review weekly.
  • Unredacted context. Prompt logs retain customer documents beyond their purpose. Debug: run a redaction test with a known sensitive string and confirm it never reaches storage.

In production

Track zero-result rate, p95 per stage and the rank of the passage a citation resolved to. Cap retention per field and alert on a drop in retrieval recall proxies after a deploy.

Common misconceptions

People think: Logging the final prompt is enough.

Actually: Without candidate ids and scores you cannot tell whether retrieval or assembly lost the evidence.

People think: Observability is only for incidents.

Actually: The same traces are the dataset for retrieval evaluation and for tuning.

Interview angle

Name the join key, the intermediate artefacts worth storing and the metrics derived from them. Treat retention and redaction as part of the schema. Logging prompts and responses, and nothing between them, is the answer that cannot locate a failure.

Sources