Skip to content
AI Engineer Guide.
midReportedcurated

How do you evaluate retrieval separately from generation, and which metric do you fix first?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Why report recall@50 before reranking and recall@8 after, what decision does each number support?
  • Recall@50 says whether the first stage found the evidence at all, which is the ceiling the reranker inherits and cannot exceed.
  • Recall@8 says whether the evidence survived the cut into the prompt, which is all the generator can use.
  • A gap between the two points at rerank depth or candidate quality, while two low numbers point at retrieval itself.
Retrieval recall goes up but answer correctness goes down, how do you isolate the cause?
  • Check whether the added evidence is the right evidence, because higher recall can also mean more distractors entering the prompt.
  • Print the assembled prompt for the regressed questions and look for contradicting passages or the right sentence buried mid-context.
  • Re-ask with retrieval removed, since an unchanged answer means the generator was never using the context.
How do you maintain chunk-level labels when re-chunking invalidates every one of them?
  • Store labels as document span offsets rather than chunk IDs, then resolve them to whatever chunk covers the span after a rebuild.
  • Keep a stable core of document-level labels for longitudinal comparison and re-derive chunk labels per chunking configuration.
  • Version the eval set with the chunker, so a score is always reported against the labels built for that configuration.

Red flags

  • blended score
  • synthetic only

Concepts this tests

Tags

Source: algorithmine.com

More in RAG & retrieval pipelines