Skip to content
AI Engineer Guide.

drill · drills · beginner · concept 13 of 33

Debug: the RAG system answers from the wrong policy

0 of 33 complete

Time box: 8 minutes · Format: verbal + whiteboard · What is tested: localisation, not prompt tweaking

What the interviewer is listening for

Whether you separate retrieval, freshness, chunking, and generation before touching a prompt. Candidates who start with “improve the system prompt” fail this drill. The strong answer narrows the search space with evidence, in order, and names the one check that would falsify each hypothesis.

A strong answer

Start from the artifact, not the answer. Print the exact prompt the model saw: the retrieved chunk IDs, their scores, and the assembled context. Three things become visible at once — which document was used, where it sits in the context, and whether a newer version existed but lost.

Then walk the pipeline backwards.

  1. Is the cited chunk current? Compare its updated_at and document version against the source of truth. If the current policy exists but was not retrieved, this is a retrieval or index problem, not a generation problem.
  2. Does the chunk contain the number? If the chunk is the old policy, retrieval is working as designed and freshness failed. If the chunk is current but the model still says 30, the context is being ignored or contradicted.
  3. Was there a conflicting chunk? Two policies in context — old and new — produce exactly this failure. Check for near-duplicate documents and missing effective-date filtering.
  4. Did chunking split the table? A refund table separated from its heading produces numbers with no scope. Read the chunk verbatim; do not trust the summary.
  5. Did the query rewrite drop the qualifier? “Refund window for annual plans” rewritten to “refund policy” widens the search and can surface the old plan.
  6. Is the index stale? Check ingestion lag for that connector. A document-ID scheme change can silently freeze a connector for days.

The decisive check is usually step 1 or 3. In this scenario the current document exists, but the old PDF was never deleted, both are indexed, and the old chunk has a slightly higher similarity because it repeats the query wording. The fix is three-part: propagate deletes, filter by effective date, and add the case to the eval set. A prompt patch would have hidden it for a week.

The trap

The data path before the promptInteractive
startquestionoutcomepath takenruled out
VerdictChoose· recommended hereConditional· with the named costAvoid· the page advises against

Each edge carries the answer read at the fork; the dashed card at the foot is the prompt patch the page rules out, not a rung.

Step 1 of 7

Start from the artifact

Print the exact prompt: chunk IDs, their scores and the assembled context. Three things become visible at once: which document was used, where it sits, and whether a newer version existed but lost.

question

Is the cited chunk current?

Compare updated_at and document version against the source of truth. If the current policy exists but was not retrieved, this is a retrieval or index problem.

Path walked

  1. The answer cites the old policy
  2. Is the cited chunk current?

Ruled out: Patch the prompt to prefer newer documents

Step 1/7 · Start from the artifact

Figure. One wrong answer, five falsifiable checks: whether the cited chunk is current, whether a second policy is in context, whether chunking split the table, whether the rewrite dropped the qualifier, and whether the connector is stale. The data path is cleared before generation is suspected; the dashed card at the foot is the prompt patch the page rules out.

What to notice. The checks are the page's own; the order clears the data path before it touches the prompt, and the page calls the cited chunk and the conflicting duplicate the decisive checks. The page's second check, whether the chunk contains the number, is folded into rung 1: a current chunk that lacks the number means the context is being ignored. No timeout, score, chunk size or similarity threshold appears because the page names none. The figure does not claim a prompt patch can never help — it claims the patch hides a data-path failure while the duplicate remains.

Text version

A decision ladder for debugging a RAG answer that cites an old policy. The start says the answer cites the old policy while retrieval looks healthy. Rung 1 asks whether the cited chunk is current; no leads to fixing the data path first, by propagating deletes, filtering by effective date and adding the case to the evaluation set. Yes continues to rung 2: is a second, older policy also in context? Yes leads to filtering by effective date, because both policies are indexed and the older one wins on wording. No continues to rung 3: did chunking split the table from its heading? Yes leads to re-chunking with the heading. No continues to rung 4: did the query rewrite drop the qualifier? Yes leads to keeping the qualifier. No continues to rung 5: is the connector's index stale? Yes leads to fixing the connector lag; no leads to inspecting generation, conditional because the right chunk was in context and the model contradicted it. A separate dashed card at the foot, marked Avoid, is the tempting prompt patch to prefer newer documents: it leaves the duplicate indexed, and the page rules it out.

The tempting fix is “tell the model to prefer newer documents.” It fails for three reasons: the model cannot see document timestamps unless you put them in context, the instruction competes with a fluent contradicting passage, and the underlying duplicate remains. Fix the data path, then make the prompt explicit about recency as a belt-and-braces measure.

Rubric

ScoreWhat it looks like
1Changes the prompt first, no evidence gathered
2Names retrieval or freshness but does not inspect the chunk
3Finds the stale duplicate but proposes only a prompt fix
4Localises to duplicate documents, fixes deletes and date filtering, adds an eval case
5Also checks connector lag and query rewriting, and explains which metric would have caught it earlier

Follow-ups

  • “How would you have caught this before a customer did?” A freshness probe: a fixed set of policy questions whose answers change on a known date, run nightly against the live index.
  • “What if the old document is still legally required?” Keep it, but attach validity dates and filter at retrieval — never rely on the model to choose the right version.
  • “Which metric moves first?” Retrieval hit rate on the policy slice, not end-to-end answer quality, because the aggregate score is dominated by easy questions.

Sources