Skip to content
AI Engineer Guide.
staffPracticecurated

Your RAG system works on 10k documents but degrades at 1M. What changes besides the index?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Which metric do you watch first as the corpus grows past 1M, and what alert threshold do you set?
  • Watch recall@k against exact search on a fixed probe set, because it is the ceiling everything downstream inherits.
  • Duplicate and near-duplicate flooding rises with growth, while result-count shortfalls track how selective the filter is rather than corpus size.
  • Log pre-filter and post-filter candidate counts per query, and alert when a returned count falls short of k.
  • Set the alert at a drop you would actually act on, measured on that same probe set after each ingest milestone.
What dedupe policy do you run, and how do you avoid collapsing legitimately similar documents?
  • Hash normalised content for exact duplicates, and use embedding-based near-duplicate detection with a threshold reviewed on samples.
  • Keep the copy with the newest effective date and link the superseded one, so a legitimate near-duplicate stays auditable.
  • Review merges by hand in the first pass, because collapsing two regional variants of a policy is data loss rather than housekeeping.
How do you refresh a golden set built on 10k documents so it represents the 1M tail?
  • Sample new cases from recent logs stratified by document type, query length and tenant, so rare strata are not drowned proportionally.
  • Keep a stable core for longitudinal comparison and add a rotating fresh sample, or the set becomes a museum of old traffic.
  • Deprecate cases whose documents no longer exist and add replacements, rather than editing history and breaking old runs.

Red flags

  • only buys a bigger index
  • no eval refresh

Concepts this tests

Tags

More in RAG & retrieval pipelines