staffPracticecurated
Your RAG system works on 10k documents but degrades at 1M. What changes besides the index?
30-second answer
- Several things degrade at once, which is why the symptom arrives as one mysterious quality drop rather than a single failing component.
- ANN recall at fixed ef falls as N grows, so re-tune against exact ground truth instead of reusing the value that worked at 10k documents.
- Duplicate and near-duplicate documents multiply, so dedupe by content hash, and a single global corpus loses precision now that the right answer is one of a million.
- Metadata filters and query routing restore precision, while ingestion becomes a pipeline problem with backfills and deletes to own.
- Eval sets built on 10k documents no longer represent the tail, and sharding or per-tenant indexes restore isolation and latency at the cost of operational overhead.
Likely follow-ups
Open a question to rehearse the answer.
Which metric do you watch first as the corpus grows past 1M, and what alert threshold do you set?
- Watch recall@k against exact search on a fixed probe set, because it is the ceiling everything downstream inherits.
- Duplicate and near-duplicate flooding rises with growth, while result-count shortfalls track how selective the filter is rather than corpus size.
- Log pre-filter and post-filter candidate counts per query, and alert when a returned count falls short of k.
- Set the alert at a drop you would actually act on, measured on that same probe set after each ingest milestone.
What dedupe policy do you run, and how do you avoid collapsing legitimately similar documents?
- Hash normalised content for exact duplicates, and use embedding-based near-duplicate detection with a threshold reviewed on samples.
- Keep the copy with the newest effective date and link the superseded one, so a legitimate near-duplicate stays auditable.
- Review merges by hand in the first pass, because collapsing two regional variants of a policy is data loss rather than housekeeping.
How do you refresh a golden set built on 10k documents so it represents the 1M tail?
- Sample new cases from recent logs stratified by document type, query length and tenant, so rare strata are not drowned proportionally.
- Keep a stable core for longitudinal comparison and add a rotating fresh sample, or the set becomes a museum of old traffic.
- Deprecate cases whose documents no longer exist and add replacements, rather than editing history and breaking old runs.
Red flags
- only buys a bigger index
- no eval refresh
Concepts this tests
Tags
More in RAG & retrieval pipelines
- Explain parent-child (small-to-big) retrieval. What problem does it solve and what does it cost you?
- Explain HNSW. What do M, ef_construction and ef_search each control, and which can you change in production?
- How would you tune HNSW for 10 million vectors when the product needs p95 retrieval under 100 ms?