midReportedcurated
How do you evaluate retrieval separately from generation, and which metric do you fix first?
30-second answer
- Evaluate the halves separately because they fail separately: retrieval decides whether the evidence arrives, generation decides what is done with it.
- Score retrieval on a golden set of real questions labelled with acceptable relevant chunks, using recall@k for arrival and nDCG or MRR for ordering.
- Report recall@50 before reranking and recall@8 after, because the reranker can only recover what stage one surfaced.
- Hold retrieval fixed, then measure faithfulness, correctness and citation accuracy on the generated answers.
- Raise recall first because it is the ceiling; chunk-level labels are expensive to maintain, while document-level labels flatter a system that fails at chunk granularity.
Likely follow-ups
Open a question to rehearse the answer.
Why report recall@50 before reranking and recall@8 after, what decision does each number support?
- Recall@50 says whether the first stage found the evidence at all, which is the ceiling the reranker inherits and cannot exceed.
- Recall@8 says whether the evidence survived the cut into the prompt, which is all the generator can use.
- A gap between the two points at rerank depth or candidate quality, while two low numbers point at retrieval itself.
Retrieval recall goes up but answer correctness goes down, how do you isolate the cause?
- Check whether the added evidence is the right evidence, because higher recall can also mean more distractors entering the prompt.
- Print the assembled prompt for the regressed questions and look for contradicting passages or the right sentence buried mid-context.
- Re-ask with retrieval removed, since an unchanged answer means the generator was never using the context.
How do you maintain chunk-level labels when re-chunking invalidates every one of them?
- Store labels as document span offsets rather than chunk IDs, then resolve them to whatever chunk covers the span after a rebuild.
- Keep a stable core of document-level labels for longitudinal comparison and re-derive chunk labels per chunking configuration.
- Version the eval set with the chunker, so a score is always reported against the labels built for that configuration.
Red flags
- blended score
- synthetic only
Concepts this tests
Tags
Source: algorithmine.com
More in RAG & retrieval pipelines
- Explain parent-child (small-to-big) retrieval. What problem does it solve and what does it cost you?
- Explain HNSW. What do M, ef_construction and ef_search each control, and which can you change in production?
- How would you tune HNSW for 10 million vectors when the product needs p95 retrieval under 100 ms?