Skip to content
AI Engineer Guide.

drill · drills · beginner · concept 20 of 33

Take-home: RAG

0 of 33 complete

Why it matters

A RAG demo can look convincing on three questions and collapse on the fourth. The take-home is designed to see whether you can build the measurement that reveals this: a labelled query set, recall at the depth you pass to the model, and a check that answers are grounded in retrieved text. Reviewers also look at the unglamorous parts, ingestion and chunking, because those decisions are hard to change later.

Mental model

It is an open-book exam: the score depends on which pages are handed over as much as on the writing. The metaphor breaks because an exam has a known textbook, while your corpus and its questions are chosen by you and must be justified.

How it works

Mechanism, step by stepInteractive
  1. 1Describe the corpus, the retrieval unit and the chunking rule, and why that unit matches the questions.
  2. 2Build a labelled set of questions with the passage that answers each, including questions with no answer.
  3. 3Measure recall at the k you pass to the model, then measure answer correctness separately.
  4. 4Require citations that resolve to a retrieved passage id, and verify them programmatically.
  5. 5Add an abstain path when retrieval returns nothing above a threshold, and test it.
  6. 6Report results per question type and list two failures with the stage where they occurred.

Step 1 of 6

Describe the corpus, the retrieval unit and the chunking rule, and why that unit matches the questions.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Describe the corpus, the retrieval unit and the chunking rule, and why that unit matches … 2. Build a labelled set of questions with the passage that answers each, including questions… 3. Measure recall at the k you pass to the model, then measure answer correctness separately. 4. Require citations that resolve to a retrieved passage id, and verify them programmaticall… 5. Add an abstain path when retrieval returns nothing above a threshold, and test it. 6. Report results per question type and list two failures with the stage where they occurred.

  1. Describe the corpus, the retrieval unit and the chunking rule, and why that unit matches the questions.
  2. Build a labelled set of questions with the passage that answers each, including questions with no answer.
  3. Measure recall at the k you pass to the model, then measure answer correctness separately.
  4. Require citations that resolve to a retrieved passage id, and verify them programmatically.
  5. Add an abstain path when retrieval returns nothing above a threshold, and test it.
  6. Report results per question type and list two failures with the stage where they occurred.

Trade-offs

ChoiceGood forCostFails when
Small chunksPrecise citationsAnswers split across boundariesQuestions need several paragraphs of context
Large chunksMore context per hitNoise and higher token costThe answer is one sentence in a long section
Strict citation requirementVerifiable answersMore abstentionsThe corpus is sparse on the topic
Always answerFewer refusalsConfident wrong answersThe domain requires accuracy over coverage
Trade-off mapInteractive

Good for

Precise citations

Cost

Answers split across boundaries

Fails when

Questions need several paragraphs of context

  1. Questions need several paragraphs of context
  2. The answer is one sentence in a long section
  3. The corpus is sparse on the topic
  4. The domain requires accuracy over coverage

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Small chunks is good for Precise citations, costs Answers split across boundaries, and fails when Questions need several paragraphs of context. Large chunks is good for More context per hit, costs Noise and higher token cost, and fails when The answer is one sentence in a long section. Strict citation requirement is good for Verifiable answers, costs More abstentions, and fails when The corpus is sparse on the topic. Always answer is good for Fewer refusals, costs Confident wrong answers, and fails when The domain requires accuracy over coverage

Failure modes

  • Retrieval never measured. End-to-end accuracy is reported with no recall number and the cause is unknown. Debug: compute recall at k on the labelled questions separately.
  • Citation not checked. Cited passage ids do not contain the claim. Debug: verify each citation resolves and that the claim appears in the cited text.
  • No abstain path. A question outside the corpus gets a plausible fabricated answer. Debug: add ten unanswerable questions and count refusals.

In production

Log retrieved ids, scores and citations per answer, and keep the labelled set runnable with one command. Alert when the abstain rate moves sharply, since it usually signals an ingestion change.

Common misconceptions

People think: A better vector database improves a RAG take-home.

Actually: Reviewers weight chunking, measurement and grounding, which are independent of the engine.

People think: End-to-end accuracy is the metric to report.

Actually: It conflates retrieval and generation, so it cannot tell a reviewer which stage to fix.

Interview angle

Strong answers separate retrieval from generation, show a labelled set, and discuss abstention and citation checks. Weak answers demo a chat window and report one accuracy number with no retrieval measurement.

Sources