Why it matters
A complaint that retrieval is irrelevant can mean four different failures: the document was never ingested, the chunk split the answer, the filter excluded it, or the ranker placed it below the cut. Each has a different fix, and similarity threshold tuning only addresses one of them. The drill tests whether you can localise the failure with evidence instead of adjusting parameters until the example looks better.
Mental model
It is a library search that returned the wrong book: the book may be missing, mis-shelved, or in your hands already. The metaphor breaks because the catalogue is approximate, so a book can be present and still not surface for a reasonable query.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Cumulative distribution of the answer's rank across 100 failing queries: the share whose answer sits at or above each retrieval depth. The 52% inside the top-10 cut were retrieved and unused, so depth is not their cause; the rising segment from 10 to 50 is the share a deeper cut could reach; the flat top is the share with no answer at any rank. Counts are illustrative.
What to notice. At the top-10 cut, 52% of the failing queries had their answer inside the cut: retrieving deeper cannot help them, so the failure is in context assembly or the prompt. Another 29% sit between rank 10 and 50 and are reachable by raising k. The last 19% have no answer at any rank — never ingested, split so no chunk holds it, or removed by a filter. Read the curve at the cut you actually use; only the rising segment is a depth problem.
Text version
A stepped cumulative curve from 0 to 100% against retrieval depth k from 0 to 50. It reaches 52% at k=10 and 81% at k=50, then stops. A vertical line marks the top-10 cut and a horizontal line marks 81%, the share with an answer in the index. A slider moves k from 1 to 50 and reads the share of failing queries with an answer at that depth; a panel splits the 100 failures into 52% inside the top-10 cut, 29% reached only by raising k, and 19% with no answer at any rank.
- Take one failing query and write the passage that should have been retrieved.
- Search the index directly for that passage id to confirm it exists and is current.
- Check the query's retrieved ids and scores, and whether the expected id appears below k.
- Inspect the filters applied to that query, since an over-restrictive filter removes correct results silently.
- If the passage is split, inspect the chunk boundaries around the answer.
- Change one stage and re-run the same query, then the full labelled set to check for regressions.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Lower similarity threshold | More recall | More noise and longer prompts | The corpus has many near-duplicates |
| Raise k | Catches misranked results | Token cost and distraction | The answer is buried far down |
| Add a reranker | Better ordering | Latency and another model to operate | The candidate set never contained the answer |
| Query rewriting | Handles vocabulary mismatch | Adds a model call and can drift | The query is already precise |
Good for
More recall
Cost
More noise and longer prompts
Fails when
The corpus has many near-duplicates
- The corpus has many near-duplicates
- The answer is buried far down
- The candidate set never contained the answer
- The query is already precise
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Lower similarity threshold is good for More recall, costs More noise and longer prompts, and fails when The corpus has many near-duplicates. Raise k is good for Catches misranked results, costs Token cost and distraction, and fails when The answer is buried far down. Add a reranker is good for Better ordering, costs Latency and another model to operate, and fails when The candidate set never contained the answer. Query rewriting is good for Handles vocabulary mismatch, costs Adds a model call and can drift, and fails when The query is already precise
Failure modes
- Missing document. The answer was never ingested or was deleted in a re-index. Debug: query the index by document id and compare counts with the source.
- Chunk boundary split. The answer spans two chunks and neither is sufficient alone. Debug: print the chunks adjacent to the expected passage and check for a sentence cut in half.
- Filter exclusion. A tenant or date filter removes the correct document. Debug: re-run the query with filters disabled and compare the result set.
In production
Log query, filters, retrieved ids and scores per answer, and sample failures into a labelled set. Alert when the retrieved count falls below k or the score distribution shifts.
Common misconceptions
People think: Irrelevant results mean the embedding model is bad.
Actually: Missing ingestion, chunk boundaries and filters cause more failures than the model.
People think: Raising k fixes recall.
Actually: It helps only if the correct passage is already ranked below the cut; it adds noise otherwise.
Interview angle
Strong answers localise the failure: indexed, ranked, filtered or assembled, and name the check for each. Weak answers tune a threshold before confirming the document exists.
Sources
- Karpukhin et al., Dense Passage Retrieval — recall as the retrieval metric and dense retrieval basics.
- pgvector — filtering behaviour and index parameters to inspect.
- Lewis et al., Retrieval-Augmented Generation — where retrieval sits in the answer pipeline.