Why it matters
Lexical retrieval cannot match a question to a passage that shares no terms with it, and cross-encoders are too slow to score a corpus. Dense retrieval takes the middle path: encode each side separately so passages can be embedded once, and train the two encoders jointly so the resulting vectors are comparable. The cost is compression — an entire passage becomes one point, and whatever the encoder did not learn to preserve is gone.
Mental model
A dual encoder is two translators who have agreed on a shared language: one translates questions, the other translates documents, and matching translations are deemed relevant. The metaphor breaks because the translators never meet, so a question about a detail the document translator considered unimportant lands far away.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. A query and ten synthetic passages in one hand-placed 2-D space. Passages are embedded before the query is seen; the query ranks them by cosine. The positions are illustrative, and the ranking is the only claim.
What to notice. The paraphrase why did the same order bill me two times ranks first at 0.996 without sharing a content word with the query, which asks about charged twice PAY-4092. The passage containing the exact code PAY-4092 ranks eighth at -0.42 and sits outside k = 5: the encoder compressed the identifier into the surrounding sentence, so dense retrieval loses the token match that lexical search would keep.
Text version
A scatter of ten labelled synthetic passages and a query, charged twice PAY-4092. The default frame uses k = 5: paraphrase is first at 1.00, statement line second at 0.99, support third, refund fourth and invoice fifth, all ringed and joined to the query. The passage labelled PAY-4092 shares the query's exact code but sits away from the query at rank eight and is marked missed. Two callouts restate the outcome: the paraphrase shares no content word with the query and was found at rank one; the code passage shares the exact token and was missed at rank eight.
- Collect or generate question-passage pairs where the passage genuinely answers the question, not merely shares its topic.
- Mine hard negatives with a first-stage retriever so the model learns fine distinctions rather than random ones.
- Train with an in-batch contrastive objective, watching for false negatives that share the answer with a different passage.
- Fix the maximum input length and chunk to it; text beyond the limit must move to another chunk rather than be truncated.
- Embed the corpus once, normalise if the metric requires it, and record model version and dimension per vector.
- Evaluate recall@k on held-out queries before and after any fine-tune, sliced by query type.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Off-the-shelf encoder | No training data or GPU time | Misses domain vocabulary and identifiers | The corpus is jargon-heavy or uses internal codes |
| Domain fine-tuned encoder | Better recall on in-domain phrasing | Labelling cost and re-embedding the corpus | Traffic drifts away from the training distribution |
| Short passages | Precise vectors and cheap indexing | Answers split across chunks | The question needs a multi-paragraph argument |
| Long passages | More context per vector | The vector averages several topics | A single fact is buried in a long section |
Good for
No training data or GPU time
Cost
Misses domain vocabulary and identifiers
Fails when
The corpus is jargon-heavy or uses internal codes
- The corpus is jargon-heavy or uses internal codes
- Traffic drifts away from the training distribution
- The question needs a multi-paragraph argument
- A single fact is buried in a long section
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Off-the-shelf encoder is good for No training data or GPU time, costs Misses domain vocabulary and identifiers, and fails when The corpus is jargon-heavy or uses internal codes. Domain fine-tuned encoder is good for Better recall on in-domain phrasing, costs Labelling cost and re-embedding the corpus, and fails when Traffic drifts away from the training distribution. Short passages is good for Precise vectors and cheap indexing, costs Answers split across chunks, and fails when The question needs a multi-paragraph argument. Long passages is good for More context per vector, costs The vector averages several topics, and fails when A single fact is buried in a long section
Failure modes
- False negatives in training. A passage that answers the question is treated as a negative because another passage was labelled positive, so the model learns to push correct answers away. Debug: score the labelled negatives with a cross-encoder and inspect the highest-scoring ones.
- Silent truncation. The encoder cuts text at its limit and the decisive sentence never enters the vector. Debug: compare the token count of stored chunks with the encoder maximum and list the chunks that exceed it.
- Index and query encoder drift. A fine-tuned query encoder is paired with an index built by the old passage encoder. Debug: embed a passage and its own question through both paths and confirm the pair is still nearest.
In production
Record encoder version, dimension and normalisation with the index, and rebuild rather than mix. Track recall on a frozen query set and the rate of queries whose top score falls below the abstain floor.
Common misconceptions
People think: Dense retrieval replaces lexical search.
Actually: It complements it: dense vectors lose exact identifiers and rare tokens, which lexical indexes preserve.
People think: More training pairs always improve the encoder.
Actually: Pairs that are topical but not answering teach the wrong boundary; negative quality matters more than pair count.
Interview angle
Cover the dual encoder, why passages can be precomputed, and how hard negatives shape the decision boundary. Name truncation and false negatives as the quiet failure modes. Candidates who describe embeddings generically, and cannot say why the passage vector never sees the query, have not understood the architecture they are proposing.
Sources
- Karpukhin et al., Dense Passage Retrieval — dual-encoder training with in-batch negatives.
- Reimers & Gurevych, Sentence-BERT — siamese sentence encoders and pooling choices.
- Thakur et al., BEIR — how dense retrieval generalises, and where it does not.