staffReportedcurated
How would you tune HNSW for 10 million vectors when the product needs p95 retrieval under 100 ms?
30-second answer
- Start from a requirement, not parameters: sample real queries, run exact brute-force search, keep the true top-k, then set a recall target of 0.95-0.99 against that ground truth.
- Choose M from the RAM budget, since M and dimension determine memory, then sweep ef_search for the lowest value that meets the target.
- Judge the sweep at p95 and p99 rather than mean latency, and re-measure after growth because recall at fixed ef degrades as N rises.
- Log the sweep so the next engineer inherits the reasoning rather than a parameter list with no provenance.
- The trade is memory for M against latency for ef, and neither of them removes the approximation.
Likely follow-ups
Open a question to rehearse the answer.
Would you quantise before or after sweeping ef_search, and how does that order change the answer?
- Sweep at full precision first, so the recall loss from compression is measured against a known baseline instead of being confounded with ef.
- Then re-sweep ef_search on the quantised index, because the same ef yields lower recall and the old value is stale.
- Keep exact rescoring available and measure how many candidates must be rescored to recover the target.
How do you keep the brute-force ground-truth sample from drifting as the corpus grows?
- Freeze a query sample with its exact top-k, then re-run exact search against the current corpus on a schedule and diff.
- Refresh the query sample from recent logs, or the ground truth measures questions the product no longer receives.
- Alert when exact and ANN disagree beyond the target, which catches both index drift and corpus change.
p95 meets the target but p99 does not, what do you change, and what does it cost in recall?
- Raise ef_search to lift the tail, and check whether the cost in p50 latency is acceptable on the same traffic.
- Look for queueing and cold replicas before touching the index, since the tail often comes from admission rather than search.
- If the tail remains, cap query-time effort and over-fetch so one expensive query cannot consume the budget.
Red flags
- tutorial defaults
- optimizes mean only
Concepts this tests
Tags
Source: ishwarj.com
More in RAG & retrieval pipelines
- Explain parent-child (small-to-big) retrieval. What problem does it solve and what does it cost you?
- Explain HNSW. What do M, ef_construction and ef_search each control, and which can you change in production?
- IVF, HNSW, product quantisation — pick an index for 500M vectors under a hard memory budget and defend it.