Why it matters
A broken classifier throws an error or shows a bad label. A broken RAG system writes a paragraph. The failure arrives inside a plausible answer, in the product's tone, sometimes with a citation attached, and the user has no way to tell that the evidence was missing. That makes these failures expensive: customers find them before monitoring does, unless the pipeline is deliberately instrumented.
The second problem is masking. If the generator ignores context and answers from its own knowledge, retrieval bugs stay invisible — the system looks fine on easy questions, and the eval set quietly rewards the generator's priors. A strict "answer only from context" instruction inverts the problem, turning every retrieval miss into a refusal. Only per-stage instrumentation escapes both.
Mental model
Think of the pipeline as a relay. The baton is the piece of evidence that answers the question, and it has to be handed from ingestion through parsing, chunking, indexing, retrieval, ranking and assembly before the generator can use it. Any hand-off can drop it.
The relay metaphor breaks because failures are not equally visible and do not compose additively. A parse error can shrink the corpus so that retrieval looks precise but incomplete; a stale duplicate can make retrieval look healthy while delivering the wrong policy. You cannot infer the broken stage from the final answer alone — you have to look at the intermediate artefacts.
How it works: the taxonomy
- 1Reproduce from the trace. Take the failing request ID and print the exact prompt: chunk IDs, text, order, scores, filters and the rewritten query. Almost every investigation starts and ends here.
- 2Check the prompt first. Is the correct passage present? If yes, the problem is assembly or generation. If no, walk backwards.
- 3Walk backwards one stage at a time. Was it in the candidate list? In the index? In the parsed text? In the source? Each answer eliminates half the pipeline.
- 4Fix the earliest broken stage. A prompt tweak on a retrieval bug ships a workaround that fails on the next query.
- 5Add the case to the eval set. A failure that is not in the eval set will return.
Step 1 of 5
Reproduce from the trace. Take the failing request ID and print the exact prompt: chunk IDs, text, order, scores, filters and the rewritten query. Almost every investigation starts and ends here.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Reproduce from the trace. Take the failing request ID and print the exact prompt: chunk I… 2. Check the prompt first. Is the correct passage present? If yes, the problem is assembly o… 3. Walk backwards one stage at a time. Was it in the candidate list? In the index? In the pa… 4. Fix the earliest broken stage. A prompt tweak on a retrieval bug ships a workaround that … 5. Add the case to the eval set. A failure that is not in the eval set will return.
Diagnose in this order. Stop at the first stage where the evidence disappears.
| Stage | Symptom | First check |
|---|---|---|
| Corpus | Nobody wrote it down; connector too narrow | Search the source system for the phrase |
| Parse | Table flattened, scanned page empty | Read the extracted text for that page |
| Chunk | Answer split across chunks, heading detached | Print the chunk verbatim, cold |
| Index | Document exists but is not retrievable | Query by exact title, then by document ID |
| Freshness | Old version wins; deletes never propagated | Diff document IDs between source and index |
| Query | Rewrite dropped a qualifier | Log the rewritten query and compare |
| Retrieval | Right passage absent from top 100 | Compute recall@k on labelled queries |
| Ranking | Right passage present but below the cut | Inspect the known-good chunk's rank |
| Assembly | Conflicts, too many passages, mid-context burial | Print the exact prompt the model saw |
| Generation | Context ignored, two passages merged | Re-ask with the same prompt minus retrieval |
| Citation | Real document, unsupported claim | Verify the quoted span and the entailment |
| Policy | Answer leaks a document the user cannot access | Audit filters, caches and side channels |
- Reproduce from the trace. Take the failing request ID and print the exact prompt: chunk IDs, text, order, scores, filters and the rewritten query. Almost every investigation starts and ends here.
- Check the prompt first. Is the correct passage present? If yes, the problem is assembly or generation. If no, walk backwards.
- Walk backwards one stage at a time. Was it in the candidate list? In the index? In the parsed text? In the source? Each answer eliminates half the pipeline.
- Fix the earliest broken stage. A prompt tweak on a retrieval bug ships a workaround that fails on the next query.
- Add the case to the eval set. A failure that is not in the eval set will return.
The diagnostic trace
from dataclasses import dataclass, field
@dataclass
class RetrievalTrace:
request_id: str
query_raw: str
query_rewritten: str | None
filters: dict
candidates: list[dict] = field(default_factory=list) # id, score, rank, source
kept_ids: list[str] = field(default_factory=list)
prompt_hash: str | None = None
answer: str | None = None
citation_checks: list[dict] = field(default_factory=list)
def diagnose(trace: RetrievalTrace, expected_doc_id: str) -> str:
"""Return the earliest stage where the expected evidence disappears."""
candidates = {c["id"]: c for c in trace.candidates}
if expected_doc_id not in candidates:
return "retrieval_or_earlier: never a candidate"
if expected_doc_id not in trace.kept_ids:
rank = candidates[expected_doc_id]["rank"]
return f"ranking: candidate at rank {rank}, cut before assembly"
if not trace.citation_checks:
return "generation_or_citation: no verification recorded"
return "assembly_or_generation: evidence present, answer unsupported"The value is not the code, it is the shape: a per-request record that lets you answer "where did the evidence die" without re-running the pipeline and hoping the bug reproduces.
Trade-offs
| Debug strategy | Good for | Cost | Fails when |
|---|---|---|---|
| Prompt tweaking | Assembly and instruction failures | Minutes, no data change | The evidence was never retrieved |
| Replay one failing trace | Fast localisation of a reported bug | Needs full request logging | The failure only appears on a rare query class |
| Label 50 real queries | Quantifying retrieval recall | Hours of annotation | Traffic is too new to sample representative queries |
| Synthetic question generation | Bootstrapping an eval set at cold start | Questions differ from real phrasing | The generator writes questions the retriever finds easy |
| Full prompt and chunk diffing | Stale data and duplicate documents | Storage and PII handling work | The corpus churns faster than you can diff |
Good for
Assembly and instruction failures
Cost
Minutes, no data change
Fails when
The evidence was never retrieved
- The evidence was never retrieved
- The failure only appears on a rare query class
- Traffic is too new to sample representative queries
- The generator writes questions the retriever finds easy
- The corpus churns faster than you can diff
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Prompt tweaking is good for Assembly and instruction failures, costs Minutes, no data change, and fails when The evidence was never retrieved. Replay one failing trace is good for Fast localisation of a reported bug, costs Needs full request logging, and fails when The failure only appears on a rare query class. Label 50 real queries is good for Quantifying retrieval recall, costs Hours of annotation, and fails when Traffic is too new to sample representative queries. Synthetic question generation is good for Bootstrapping an eval set at cold start, costs Questions differ from real phrasing, and fails when The generator writes questions the retriever finds easy. Full prompt and chunk diffing is good for Stale data and duplicate documents, costs Storage and PII handling work, and fails when The corpus churns faster than you can diff
Failure modes
- Silent retrieval miss. The answer is confident and wrong; the correct passage never made it into the shortlist. Debug: label the supporting document for 50 real questions, compute recall@10 and recall@100, then inspect the misses' chunk text — the usual causes are chunk boundaries and vocabulary mismatch.
- Stale duplicate. Two versions of a document are indexed and the older one has better lexical overlap with the query. Debug: search by the document title and count versions; add effective-date filtering and propagate deletes.
- Context conflict. Both versions are in the prompt and the model blends them, producing a number that appears in neither. Debug: print the prompt with source dates; if two passages disagree, that is an assembly bug, not a generation bug.
- Prompt-induced over-confidence. The instruction says "answer using the context" and the model treats weak evidence as sufficient. Debug: remove retrieval entirely and compare answers; if they are similar, the model is not using the context and your evals are measuring priors.
In production
Log one record per request: rewritten query, filters, candidate IDs with scores, kept IDs, prompt hash, output, citation verification results. Hash the text rather than storing raw PII where you can, and set a retention window that matches your compliance obligations.
Alert on metrics that move before answers degrade: abstain rate, recall@k on a fixed probe set, index freshness per connector, citation verification failure rate, and the share of answers with zero verified citations. Then re-run a nightly probe set with questions whose answers change on known dates — that is the only reliable detector for a connector that silently stopped.
Common misconceptions
People think: When RAG answers badly, improve the prompt.
Actually: Prompt changes only affect assembly and generation. If the passage was never retrieved, a better prompt makes the wrong answer more persuasive.
People think: A healthy retrieval hit rate means the pipeline is fine.
Actually: Hit rate is usually measured on questions the system already handles. Slice it by document type, query length and recency, or it hides the failures that matter.
Interview angle
The classic prompt is "the answer is wrong, walk me through your diagnosis". Interviewers are listening for ordered elimination: reproduce from the trace, check whether the evidence was in the prompt, then walk backwards. Naming a specific check per hypothesis — document-ID diff for freshness, rank inspection for ranking, prompt-free generation for context use — is what separates a senior answer from a list of things you could tweak. See debugging a bad RAG answer and RAG interview questions.
Sources
- Barnett et al., Seven Failure Points When Engineering a Retrieval Augmented Generation System — a case-study taxonomy of where RAG systems break in practice.
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — the baseline architecture these failures are described against.
- Liu et al., Lost in the Middle — position effects that cause assembly-stage failures.
- Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation — separating retrieval and generation quality in measurement.