Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 41 of 57

RAG failures

0 of 57 complete

Why it matters

A broken classifier throws an error or shows a bad label. A broken RAG system writes a paragraph. The failure arrives inside a plausible answer, in the product's tone, sometimes with a citation attached, and the user has no way to tell that the evidence was missing. That makes these failures expensive: customers find them before monitoring does, unless the pipeline is deliberately instrumented.

The second problem is masking. If the generator ignores context and answers from its own knowledge, retrieval bugs stay invisible — the system looks fine on easy questions, and the eval set quietly rewards the generator's priors. A strict "answer only from context" instruction inverts the problem, turning every retrieval miss into a refusal. Only per-stage instrumentation escapes both.

Mental model

Think of the pipeline as a relay. The baton is the piece of evidence that answers the question, and it has to be handed from ingestion through parsing, chunking, indexing, retrieval, ranking and assembly before the generator can use it. Any hand-off can drop it.

The relay metaphor breaks because failures are not equally visible and do not compose additively. A parse error can shrink the corpus so that retrieval looks precise but incomplete; a stale duplicate can make retrieval look healthy while delivering the wrong policy. You cannot infer the broken stage from the final answer alone — you have to look at the intermediate artefacts.

How it works: the taxonomy

Mechanism, step by stepInteractive
  1. 1Reproduce from the trace. Take the failing request ID and print the exact prompt: chunk IDs, text, order, scores, filters and the rewritten query. Almost every investigation starts and ends here.
  2. 2Check the prompt first. Is the correct passage present? If yes, the problem is assembly or generation. If no, walk backwards.
  3. 3Walk backwards one stage at a time. Was it in the candidate list? In the index? In the parsed text? In the source? Each answer eliminates half the pipeline.
  4. 4Fix the earliest broken stage. A prompt tweak on a retrieval bug ships a workaround that fails on the next query.
  5. 5Add the case to the eval set. A failure that is not in the eval set will return.

Step 1 of 5

Reproduce from the trace. Take the failing request ID and print the exact prompt: chunk IDs, text, order, scores, filters and the rewritten query. Almost every investigation starts and ends here.

Step 1/5 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Reproduce from the trace. Take the failing request ID and print the exact prompt: chunk I… 2. Check the prompt first. Is the correct passage present? If yes, the problem is assembly o… 3. Walk backwards one stage at a time. Was it in the candidate list? In the index? In the pa… 4. Fix the earliest broken stage. A prompt tweak on a retrieval bug ships a workaround that … 5. Add the case to the eval set. A failure that is not in the eval set will return.

Diagnose in this order. Stop at the first stage where the evidence disappears.

StageSymptomFirst check
CorpusNobody wrote it down; connector too narrowSearch the source system for the phrase
ParseTable flattened, scanned page emptyRead the extracted text for that page
ChunkAnswer split across chunks, heading detachedPrint the chunk verbatim, cold
IndexDocument exists but is not retrievableQuery by exact title, then by document ID
FreshnessOld version wins; deletes never propagatedDiff document IDs between source and index
QueryRewrite dropped a qualifierLog the rewritten query and compare
RetrievalRight passage absent from top 100Compute recall@k on labelled queries
RankingRight passage present but below the cutInspect the known-good chunk's rank
AssemblyConflicts, too many passages, mid-context burialPrint the exact prompt the model saw
GenerationContext ignored, two passages mergedRe-ask with the same prompt minus retrieval
CitationReal document, unsupported claimVerify the quoted span and the entailment
PolicyAnswer leaks a document the user cannot accessAudit filters, caches and side channels
  1. Reproduce from the trace. Take the failing request ID and print the exact prompt: chunk IDs, text, order, scores, filters and the rewritten query. Almost every investigation starts and ends here.
  2. Check the prompt first. Is the correct passage present? If yes, the problem is assembly or generation. If no, walk backwards.
  3. Walk backwards one stage at a time. Was it in the candidate list? In the index? In the parsed text? In the source? Each answer eliminates half the pipeline.
  4. Fix the earliest broken stage. A prompt tweak on a retrieval bug ships a workaround that fails on the next query.
  5. Add the case to the eval set. A failure that is not in the eval set will return.

The diagnostic trace

from dataclasses import dataclass, field
 
@dataclass
class RetrievalTrace:
    request_id: str
    query_raw: str
    query_rewritten: str | None
    filters: dict
    candidates: list[dict] = field(default_factory=list)   # id, score, rank, source
    kept_ids: list[str] = field(default_factory=list)
    prompt_hash: str | None = None
    answer: str | None = None
    citation_checks: list[dict] = field(default_factory=list)
 
def diagnose(trace: RetrievalTrace, expected_doc_id: str) -> str:
    """Return the earliest stage where the expected evidence disappears."""
    candidates = {c["id"]: c for c in trace.candidates}
    if expected_doc_id not in candidates:
        return "retrieval_or_earlier: never a candidate"
    if expected_doc_id not in trace.kept_ids:
        rank = candidates[expected_doc_id]["rank"]
        return f"ranking: candidate at rank {rank}, cut before assembly"
    if not trace.citation_checks:
        return "generation_or_citation: no verification recorded"
    return "assembly_or_generation: evidence present, answer unsupported"

The value is not the code, it is the shape: a per-request record that lets you answer "where did the evidence die" without re-running the pipeline and hoping the bug reproduces.

Trade-offs

Debug strategyGood forCostFails when
Prompt tweakingAssembly and instruction failuresMinutes, no data changeThe evidence was never retrieved
Replay one failing traceFast localisation of a reported bugNeeds full request loggingThe failure only appears on a rare query class
Label 50 real queriesQuantifying retrieval recallHours of annotationTraffic is too new to sample representative queries
Synthetic question generationBootstrapping an eval set at cold startQuestions differ from real phrasingThe generator writes questions the retriever finds easy
Full prompt and chunk diffingStale data and duplicate documentsStorage and PII handling workThe corpus churns faster than you can diff
Trade-off mapInteractive

Good for

Assembly and instruction failures

Cost

Minutes, no data change

Fails when

The evidence was never retrieved

  1. The evidence was never retrieved
  2. The failure only appears on a rare query class
  3. Traffic is too new to sample representative queries
  4. The generator writes questions the retriever finds easy
  5. The corpus churns faster than you can diff

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Prompt tweaking is good for Assembly and instruction failures, costs Minutes, no data change, and fails when The evidence was never retrieved. Replay one failing trace is good for Fast localisation of a reported bug, costs Needs full request logging, and fails when The failure only appears on a rare query class. Label 50 real queries is good for Quantifying retrieval recall, costs Hours of annotation, and fails when Traffic is too new to sample representative queries. Synthetic question generation is good for Bootstrapping an eval set at cold start, costs Questions differ from real phrasing, and fails when The generator writes questions the retriever finds easy. Full prompt and chunk diffing is good for Stale data and duplicate documents, costs Storage and PII handling work, and fails when The corpus churns faster than you can diff

Failure modes

  • Silent retrieval miss. The answer is confident and wrong; the correct passage never made it into the shortlist. Debug: label the supporting document for 50 real questions, compute recall@10 and recall@100, then inspect the misses' chunk text — the usual causes are chunk boundaries and vocabulary mismatch.
  • Stale duplicate. Two versions of a document are indexed and the older one has better lexical overlap with the query. Debug: search by the document title and count versions; add effective-date filtering and propagate deletes.
  • Context conflict. Both versions are in the prompt and the model blends them, producing a number that appears in neither. Debug: print the prompt with source dates; if two passages disagree, that is an assembly bug, not a generation bug.
  • Prompt-induced over-confidence. The instruction says "answer using the context" and the model treats weak evidence as sufficient. Debug: remove retrieval entirely and compare answers; if they are similar, the model is not using the context and your evals are measuring priors.

In production

Log one record per request: rewritten query, filters, candidate IDs with scores, kept IDs, prompt hash, output, citation verification results. Hash the text rather than storing raw PII where you can, and set a retention window that matches your compliance obligations.

Alert on metrics that move before answers degrade: abstain rate, recall@k on a fixed probe set, index freshness per connector, citation verification failure rate, and the share of answers with zero verified citations. Then re-run a nightly probe set with questions whose answers change on known dates — that is the only reliable detector for a connector that silently stopped.

Common misconceptions

People think: When RAG answers badly, improve the prompt.

Actually: Prompt changes only affect assembly and generation. If the passage was never retrieved, a better prompt makes the wrong answer more persuasive.

People think: A healthy retrieval hit rate means the pipeline is fine.

Actually: Hit rate is usually measured on questions the system already handles. Slice it by document type, query length and recency, or it hides the failures that matter.

Interview angle

The classic prompt is "the answer is wrong, walk me through your diagnosis". Interviewers are listening for ordered elimination: reproduce from the trace, check whether the evidence was in the prompt, then walk backwards. Naming a specific check per hypothesis — document-ID diff for freshness, rank inspection for ranking, prompt-free generation for context use — is what separates a senior answer from a list of things you could tweak. See debugging a bad RAG answer and RAG interview questions.

Sources