Time box: 8 minutes · Format: verbal + whiteboard · What is tested: localisation, not prompt tweaking
What the interviewer is listening for
Whether you separate retrieval, freshness, chunking, and generation before touching a prompt. Candidates who start with “improve the system prompt” fail this drill. The strong answer narrows the search space with evidence, in order, and names the one check that would falsify each hypothesis.
A strong answer
Start from the artifact, not the answer. Print the exact prompt the model saw: the retrieved chunk IDs, their scores, and the assembled context. Three things become visible at once — which document was used, where it sits in the context, and whether a newer version existed but lost.
Then walk the pipeline backwards.
- Is the cited chunk current? Compare its
updated_atand document version against the source of truth. If the current policy exists but was not retrieved, this is a retrieval or index problem, not a generation problem. - Does the chunk contain the number? If the chunk is the old policy, retrieval is working as designed and freshness failed. If the chunk is current but the model still says 30, the context is being ignored or contradicted.
- Was there a conflicting chunk? Two policies in context — old and new — produce exactly this failure. Check for near-duplicate documents and missing effective-date filtering.
- Did chunking split the table? A refund table separated from its heading produces numbers with no scope. Read the chunk verbatim; do not trust the summary.
- Did the query rewrite drop the qualifier? “Refund window for annual plans” rewritten to “refund policy” widens the search and can surface the old plan.
- Is the index stale? Check ingestion lag for that connector. A document-ID scheme change can silently freeze a connector for days.
The decisive check is usually step 1 or 3. In this scenario the current document exists, but the old PDF was never deleted, both are indexed, and the old chunk has a slightly higher similarity because it repeats the query wording. The fix is three-part: propagate deletes, filter by effective date, and add the case to the eval set. A prompt patch would have hidden it for a week.
The trap
Each edge carries the answer read at the fork; the dashed card at the foot is the prompt patch the page rules out, not a rung.
Step 1 of 7
Start from the artifact
Print the exact prompt: chunk IDs, their scores and the assembled context. Three things become visible at once: which document was used, where it sits, and whether a newer version existed but lost.
question
Is the cited chunk current?
Compare updated_at and document version against the source of truth. If the current policy exists but was not retrieved, this is a retrieval or index problem.
Path walked
- The answer cites the old policy
- Is the cited chunk current?
Ruled out: Patch the prompt to prefer newer documents
Figure. One wrong answer, five falsifiable checks: whether the cited chunk is current, whether a second policy is in context, whether chunking split the table, whether the rewrite dropped the qualifier, and whether the connector is stale. The data path is cleared before generation is suspected; the dashed card at the foot is the prompt patch the page rules out.
What to notice. The checks are the page's own; the order clears the data path before it touches the prompt, and the page calls the cited chunk and the conflicting duplicate the decisive checks. The page's second check, whether the chunk contains the number, is folded into rung 1: a current chunk that lacks the number means the context is being ignored. No timeout, score, chunk size or similarity threshold appears because the page names none. The figure does not claim a prompt patch can never help — it claims the patch hides a data-path failure while the duplicate remains.
Text version
A decision ladder for debugging a RAG answer that cites an old policy. The start says the answer cites the old policy while retrieval looks healthy. Rung 1 asks whether the cited chunk is current; no leads to fixing the data path first, by propagating deletes, filtering by effective date and adding the case to the evaluation set. Yes continues to rung 2: is a second, older policy also in context? Yes leads to filtering by effective date, because both policies are indexed and the older one wins on wording. No continues to rung 3: did chunking split the table from its heading? Yes leads to re-chunking with the heading. No continues to rung 4: did the query rewrite drop the qualifier? Yes leads to keeping the qualifier. No continues to rung 5: is the connector's index stale? Yes leads to fixing the connector lag; no leads to inspecting generation, conditional because the right chunk was in context and the model contradicted it. A separate dashed card at the foot, marked Avoid, is the tempting prompt patch to prefer newer documents: it leaves the duplicate indexed, and the page rules it out.
The tempting fix is “tell the model to prefer newer documents.” It fails for three reasons: the model cannot see document timestamps unless you put them in context, the instruction competes with a fluent contradicting passage, and the underlying duplicate remains. Fix the data path, then make the prompt explicit about recency as a belt-and-braces measure.
Rubric
| Score | What it looks like |
|---|---|
| 1 | Changes the prompt first, no evidence gathered |
| 2 | Names retrieval or freshness but does not inspect the chunk |
| 3 | Finds the stale duplicate but proposes only a prompt fix |
| 4 | Localises to duplicate documents, fixes deletes and date filtering, adds an eval case |
| 5 | Also checks connector lag and query rewriting, and explains which metric would have caught it earlier |
Follow-ups
- “How would you have caught this before a customer did?” A freshness probe: a fixed set of policy questions whose answers change on a known date, run nightly against the live index.
- “What if the old document is still legally required?” Keep it, but attach validity dates and filter at retrieval — never rely on the model to choose the right version.
- “Which metric moves first?” Retrieval hit rate on the policy slice, not end-to-end answer quality, because the aggregate score is dominated by easy questions.
Related
-
RAG interview bank — questions that drill this failure mode.
Sources
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — the pipeline this drill debugs.
- Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation — retrieval and faithfulness metrics used to catch the regression.