staffReportedcurated
Your offline recall is 0.91 but production recall is 0.40. What went wrong in the evaluation, and how do you fix it?
30-second answer
- The eval is measuring a corpus the live system never sees: hand-picked gold chunks or questions generated per chunk grade the retriever against perfect retrieval it never performs.
- Two other causes to rule out: the eval ran with different filters or k than production, and production traffic is the messy long tail rather than the phrasing the synthetic set used.
- Fix it by harvesting gold contexts from real production retrieval runs and sampling queries from logs.
- Reproduce the gap by replaying one logged production query through the offline harness and diffing the candidate lists stage by stage.
- Real logs are noisier to label, but they are the only honest signal, and a synthetic set will keep reporting a number production never sees.
Likely follow-ups
Open a question to rehearse the answer.
How do you generate synthetic questions without leaking the gold chunk into the query?
- Generate each question from the document with the target chunk hidden, so the phrasing cannot reuse that chunk's own words.
- Review twenty by hand, because questions that quote chunk phrasing verbatim are trivially retrievable and inflate both retrieval and answer scores.
- Calibrate synthetic cases against a real-data subset, and never let synthetic cases be the only gate.
What is the minimum effect size you would trust on a golden set of this size?
- With a few hundred cases, a couple of points sits inside sampling noise, so require an interval that excludes zero before claiming a win.
- Report the interval per slice, because an aggregate interval on a mixed set hides a real regression in a small stratum.
- Treat the number as a regression detector rather than a precise estimate, and confirm large claims on a holdout.
Which production filter or k mismatch would you check first, and how do you reproduce it offline?
- Check k first, because comparing an offline run at 50 against production at 8 flatters recall directly.
- Then check the filters: tenant, version and recency predicates omitted offline turn a restricted corpus into an easier problem.
- Replay a logged production request through the harness and diff the candidate lists, which localises the mismatch in one run.
Red flags
- concludes a bug
- bigger model
Concepts this tests
Tags
Source: github.com
More in RAG & retrieval pipelines
- Explain parent-child (small-to-big) retrieval. What problem does it solve and what does it cost you?
- Explain HNSW. What do M, ef_construction and ef_search each control, and which can you change in production?
- How would you tune HNSW for 10 million vectors when the product needs p95 retrieval under 100 ms?