Why it matters
Users rarely type the words that appear in the document that answers them. Lexical matching fails on that gap, and semantic search closes it by mapping both sides into a space where closeness means similarity of meaning. The engineering cost is that the mapping is learned, opaque and fixed at index time: every document is compressed into one vector before anyone knows what will be asked.
Mental model
Semantic search is a map of a city where distance means topical relatedness rather than geography. Neighbourhoods of similar documents cluster, and a query is a point dropped onto the map. The metaphor breaks at the edges: two places can be adjacent on the map while one is the answer and the other is merely nearby.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. One query and twelve synthetic passages in one hand-placed space. Documents are embedded before the query is seen; the query retrieves its nearest passages by cosine. Positions are illustrative; the ranking is the only claim.
What to notice. The query shares no content word with refund policy, which ranks first. A passage that does share how, do, I and my — billing address — sits outside k: lexical overlap is not meaning. Dense similarity is topical, so money-back guarantee ranks second without being the answer for a specific order. The callouts show the success and the lexical near-miss side by side.
Text version
A scatter of twelve synthetic passages and one query, how do I get my money back. The default frame uses k = 3: refund policy ranks first, money-back guarantee second and returns process third, all ringed and joined to the query. Refund policy shares no content word with the query. A billing address passage, which shares how, do, I and my with the query, sits outside k and is marked missed. Two callouts restate the semantic hit and the lexical near-miss. A slider sets k from one to five; three step controls show documents, query and neighbours.
- Choose the embedding model and fix the preprocessing, including truncation length and normalisation, before indexing anything.
- Embed documents in batches, store the vector with its model identifier and dimension, and build an approximate index over it.
- Embed the query with the same model and preprocessing path, then retrieve the nearest neighbours.
- Calibrate a similarity floor on labelled pairs so unrelated queries can return nothing instead of the least-bad passage.
- Inspect the top results for a sample of real queries and classify misses as vocabulary, granularity or ranking failures.
- Compare against a lexical baseline on the same judged set before replacing anything.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Dense-only retrieval | Paraphrase and cross-lingual matching | Misses exact identifiers and rare terms | Queries contain codes, names or numeric constraints |
| Low similarity floor | High recall | Irrelevant passages reach the generator | The corpus is heterogeneous and queries are narrow |
| High similarity floor | Fewer distractors and an abstain path | Legitimate paraphrases get filtered out | The embedding model is weak on the domain |
| Large chunks | Enough context per hit | The vector averages several topics | Answers live in one sentence of a long section |
Good for
Paraphrase and cross-lingual matching
Cost
Misses exact identifiers and rare terms
Fails when
Queries contain codes, names or numeric constraints
- Queries contain codes, names or numeric constraints
- The corpus is heterogeneous and queries are narrow
- The embedding model is weak on the domain
- Answers live in one sentence of a long section
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Dense-only retrieval is good for Paraphrase and cross-lingual matching, costs Misses exact identifiers and rare terms, and fails when Queries contain codes, names or numeric constraints. Low similarity floor is good for High recall, costs Irrelevant passages reach the generator, and fails when The corpus is heterogeneous and queries are narrow. High similarity floor is good for Fewer distractors and an abstain path, costs Legitimate paraphrases get filtered out, and fails when The embedding model is weak on the domain. Large chunks is good for Enough context per hit, costs The vector averages several topics, and fails when Answers live in one sentence of a long section
Failure modes
- Threshold copied from another corpus. Either everything passes or nothing does, and the team blames the model. Debug: plot the score distribution of labelled relevant and irrelevant pairs and pick a floor from the overlap.
- Topic match mistaken for answer. The retrieved passage is about the right product but the wrong plan tier. Debug: read the top three passages and check whether the distinguishing qualifier is present.
- Query and document preprocessing diverge. Queries are truncated or lowercased differently from documents, so distances shift. Debug: embed one known query-document pair through both paths and compare the resulting similarity.
In production
Monitor the distribution of top-1 similarity and the share of queries below the abstain floor. Re-embed and re-calibrate whenever the model version changes, and keep a lexical fallback for identifier queries.
Common misconceptions
People think: Semantic search understands the question.
Actually: It measures similarity between two texts; it has no notion of which text answers which question.
People think: A similarity score is a confidence.
Actually: The score is relative to the corpus and the model, so a 0.8 on one index can mean something entirely different on another.
Interview angle
The crux is the bi-encoder trade-off: documents are embedded without the query, so ranking is topical rather than answer-level. Discuss calibrating the similarity floor and the identifier failure that pushes teams to hybrid search. Saying the model understands meaning, without explaining why an exact part number is missed, is the usual weak answer.
Sources
- Reimers & Gurevych, Sentence-BERT — the bi-encoder architecture that makes semantic search precomputable.
- Karpukhin et al., Dense Passage Retrieval — dual-encoder retrieval trained on question-passage pairs.
- Thakur et al., BEIR — evidence that semantic retrieval quality varies by domain and query type.