Skip to content
AI Engineer Guide.
seniorReportedcurated

Your recall@5 is 95%. Does that mean the system answers 95% of questions correctly?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

What is the answer-quality ceiling implied by imperfect relevance labels, and how do you estimate it?
  • Labels bound the measurement: a chunk you failed to label relevant counts as a miss, so measured recall is capped by how completely the set was labelled.
  • Double-labelling measures label noise rather than truth, so report it as an uncertainty band around the recall figure and state the judged fraction per query.
  • Unjudged chunks count as irrelevant, so a thin judged fraction drags the score down arbitrarily; re-judge a sample when the corpus or query mix changes.
  • Adjudicate disagreements and revise the guideline rather than only the label, since annotation drift is a real source of evaluation drift.
Recall is high but correctness is low, how do you tell retrieval, context ordering and generation apart?
  • Hold retrieval fixed and re-ask with the same prompt minus retrieval, because an unchanged answer means the generator never used the context.
  • Move the winning passage to the end of the evidence block and re-run; if the answer changes, the problem is ordering.
  • If neither helps, inspect assembly for contradicting passages and truncation before blaming the model.
Which single change would you try first to close a 15-point recall-to-correctness gap, and how do you measure it?
  • Start with the instruction and abstain path: require a citation per factual sentence and an explicit refusal when the context is insufficient.
  • Then cap how many passages enter the prompt and order the strongest evidence last, since distractor volume is the usual reason a correct passage goes unused.
  • Measure with claim-level faithfulness on the same questions, so the fix is judged on the gap rather than on the retrieval metric.

Red flags

  • says yes
  • treats metrics as interchangeable

Concepts this tests

Tags

Source: ishwarj.com

More in RAG & retrieval pipelines