Skip to content
AI Engineer Guide.
staffReportedcurated

Your offline recall is 0.91 but production recall is 0.40. What went wrong in the evaluation, and how do you fix it?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

How do you generate synthetic questions without leaking the gold chunk into the query?
  • Generate each question from the document with the target chunk hidden, so the phrasing cannot reuse that chunk's own words.
  • Review twenty by hand, because questions that quote chunk phrasing verbatim are trivially retrievable and inflate both retrieval and answer scores.
  • Calibrate synthetic cases against a real-data subset, and never let synthetic cases be the only gate.
What is the minimum effect size you would trust on a golden set of this size?
  • With a few hundred cases, a couple of points sits inside sampling noise, so require an interval that excludes zero before claiming a win.
  • Report the interval per slice, because an aggregate interval on a mixed set hides a real regression in a small stratum.
  • Treat the number as a regression detector rather than a precise estimate, and confirm large claims on a holdout.
Which production filter or k mismatch would you check first, and how do you reproduce it offline?
  • Check k first, because comparing an offline run at 50 against production at 8 flatters recall directly.
  • Then check the filters: tenant, version and recency predicates omitted offline turn a restricted corpus into an easier problem.
  • Replay a logged production request through the harness and diff the candidate lists, which localises the mismatch in one run.

Red flags

  • concludes a bug
  • bigger model

Concepts this tests

Tags

Source: github.com

More in RAG & retrieval pipelines