seniorReportedcurated
Your recall@5 is 95%. Does that mean the system answers 95% of questions correctly?
30-second answer
- No: recall@5 says that for 95% of your questions a chunk you labelled relevant appeared in the top five, on your eval set, under your labels.
- It says nothing about what the generator did with it, since the model can use half the evidence, be derailed by a contradicting chunk, lose the fact mid-context, or answer from parametric memory.
- Retrieval recall is an upper bound, and the gap to correctness is generation work rather than index work.
- Treat any specific recall-and-correctness pair as illustrative unless you measured it on your own harness, because the size of the gap is what you own.
- Closing the gap costs generation changes and fresh labels, not a bigger index.
Likely follow-ups
Open a question to rehearse the answer.
What is the answer-quality ceiling implied by imperfect relevance labels, and how do you estimate it?
- Labels bound the measurement: a chunk you failed to label relevant counts as a miss, so measured recall is capped by how completely the set was labelled.
- Double-labelling measures label noise rather than truth, so report it as an uncertainty band around the recall figure and state the judged fraction per query.
- Unjudged chunks count as irrelevant, so a thin judged fraction drags the score down arbitrarily; re-judge a sample when the corpus or query mix changes.
- Adjudicate disagreements and revise the guideline rather than only the label, since annotation drift is a real source of evaluation drift.
Recall is high but correctness is low, how do you tell retrieval, context ordering and generation apart?
- Hold retrieval fixed and re-ask with the same prompt minus retrieval, because an unchanged answer means the generator never used the context.
- Move the winning passage to the end of the evidence block and re-run; if the answer changes, the problem is ordering.
- If neither helps, inspect assembly for contradicting passages and truncation before blaming the model.
Which single change would you try first to close a 15-point recall-to-correctness gap, and how do you measure it?
- Start with the instruction and abstain path: require a citation per factual sentence and an explicit refusal when the context is insufficient.
- Then cap how many passages enter the prompt and order the strongest evidence last, since distractor volume is the usual reason a correct passage goes unused.
- Measure with claim-level faithfulness on the same questions, so the fix is judged on the gap rather than on the retrieval metric.
Red flags
- says yes
- treats metrics as interchangeable
Concepts this tests
Tags
Source: ishwarj.com
More in RAG & retrieval pipelines
- Explain parent-child (small-to-big) retrieval. What problem does it solve and what does it cost you?
- Explain HNSW. What do M, ef_construction and ef_search each control, and which can you change in production?
- How would you tune HNSW for 10 million vectors when the product needs p95 retrieval under 100 ms?