Skip to content
AI Engineer Guide.

drill · drills · beginner · concept 19 of 33

Code reading: retrieval

0 of 33 complete

Why it matters

Retrieval systems fail in places that are invisible in the architecture diagram: a chunk boundary splits an answer, a filter is applied after the search, a normalisation step differs between indexing and querying. Reading the code means tracing one query through every stage and checking that the same preprocessing is applied on both sides. Reviewers who start at the vector database usually stop before the stage that caused the problem.

Mental model

It is a postal sorting line: text is cut into parcels, labelled, sorted into bins and then ranked for delivery. The metaphor breaks because a mis-sorted parcel is still delivered somewhere plausible, so the error surfaces as a bad answer rather than a lost item.

How it works

The filter, not the rank, sets the countInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Share of queries that return at least k of the ten requested results, with a filter applied after the search against the same filter applied inside it. The counts follow a binomial model; the loss of result count is the claim.

What to notice. At 70% selectivity a post-filter keeps a median of seven results: only 2.8% of queries come back with all ten and 4.7% with fewer than five. At 30% selectivity the median falls to three and 85% of queries return fewer than five. The index still returns ten; the code that filters afterwards decides what the caller sees.

Text version

Share of queries that return at least k results, against k from zero to ten. The pre-filter series is a flat line at 100% for every k. The post-filter series at 70% selectivity falls from 100% at k=1 to 95.3% at k=5, 65% at k=7 and 2.8% at k=10, where the line holds to the plot edge, with a marked median of seven. Two more options set selectivity to 50% and 30%: at 30% the curve falls to 61.7% at k=3 and under 0.1% at k=10, and the median marker moves to three. A panel reports the median results kept, the share of queries keeping all ten, and the share with fewer than five.

  1. Find the ingestion path and note the chunk size, overlap and any preprocessing such as HTML stripping or header injection.
  2. Confirm that query text goes through the same normalisation and embedding model as the documents.
  3. Read the index call for top-k, distance metric and filter placement, since post-filtering silently returns fewer results.
  4. Check whether a reranker or score fusion changes the order, and on what candidate set.
  5. Locate the context assembly step, because deduplication and token limits decide what the model actually sees.
  6. Find the evaluation code and the labelled queries; without them, note that recall is unmeasured.

Trade-offs

ChoiceGood forCostFails when
Fixed-size chunksPredictable and simpleSplits answers across boundariesDocuments have long structured sections
Semantic chunkingBetter boundariesVariable size and slower ingestionThe splitter model is unavailable at query time
Post-filter then rerankSimple pipelineFewer than k results after filteringFilters are selective
Pre-filter in the indexCorrect result countsIndex support varies by engineThe engine only supports approximate post-filtering
Trade-off mapInteractive

Good for

Predictable and simple

Cost

Splits answers across boundaries

Fails when

Documents have long structured sections

  1. Documents have long structured sections
  2. The splitter model is unavailable at query time
  3. Filters are selective
  4. The engine only supports approximate post-filtering

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Fixed-size chunks is good for Predictable and simple, costs Splits answers across boundaries, and fails when Documents have long structured sections. Semantic chunking is good for Better boundaries, costs Variable size and slower ingestion, and fails when The splitter model is unavailable at query time. Post-filter then rerank is good for Simple pipeline, costs Fewer than k results after filtering, and fails when Filters are selective. Pre-filter in the index is good for Correct result counts, costs Index support varies by engine, and fails when The engine only supports approximate post-filtering

Failure modes

  • Query and document mismatch. Queries are embedded raw while documents were embedded with a title prefix. Debug: log the exact text sent to the embedding model on both paths.
  • Silent empty filter result. A filtered query returns three results and the prompt is built from them anyway. Debug: log result count before and after filtering and alert below k.
  • Stale index. Documents were re-chunked but the index still holds old vectors. Debug: compare the document count and chunk hash in the index against the source.

In production

Keep a trace of chunk ids and scores for every generated answer, and sample queries to measure recall against a labelled set. Alert when retrieved context falls below the expected count.

Common misconceptions

People think: The vector database is the retrieval system.

Actually: It is one stage; chunking, preprocessing and filtering usually decide quality.

People think: Better embeddings fix bad retrieval.

Actually: They cannot recover an answer that was split across chunks or excluded by a filter.

Interview angle

Trace a query end to end, name where recall can be lost, and say which stage you would instrument first. Mention the evaluation set. Weak answers start at the index and never inspect how text was chunked or filtered.

Sources