Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 5 of 57

Semantic search

0 of 57 complete

Why it matters

Users rarely type the words that appear in the document that answers them. Lexical matching fails on that gap, and semantic search closes it by mapping both sides into a space where closeness means similarity of meaning. The engineering cost is that the mapping is learned, opaque and fixed at index time: every document is compressed into one vector before anyone knows what will be asked.

Mental model

Semantic search is a map of a city where distance means topical relatedness rather than geography. Neighbourhoods of similar documents cluster, and a query is a point dropped onto the map. The metaphor breaks at the edges: two places can be adjacent on the map while one is the answer and the other is merely nearby.

How it works

One query · neighbours by meaningInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. One query and twelve synthetic passages in one hand-placed space. Documents are embedded before the query is seen; the query retrieves its nearest passages by cosine. Positions are illustrative; the ranking is the only claim.

What to notice. The query shares no content word with refund policy, which ranks first. A passage that does share how, do, I and my — billing address — sits outside k: lexical overlap is not meaning. Dense similarity is topical, so money-back guarantee ranks second without being the answer for a specific order. The callouts show the success and the lexical near-miss side by side.

Text version

A scatter of twelve synthetic passages and one query, how do I get my money back. The default frame uses k = 3: refund policy ranks first, money-back guarantee second and returns process third, all ringed and joined to the query. Refund policy shares no content word with the query. A billing address passage, which shares how, do, I and my with the query, sits outside k and is marked missed. Two callouts restate the semantic hit and the lexical near-miss. A slider sets k from one to five; three step controls show documents, query and neighbours.

  1. Choose the embedding model and fix the preprocessing, including truncation length and normalisation, before indexing anything.
  2. Embed documents in batches, store the vector with its model identifier and dimension, and build an approximate index over it.
  3. Embed the query with the same model and preprocessing path, then retrieve the nearest neighbours.
  4. Calibrate a similarity floor on labelled pairs so unrelated queries can return nothing instead of the least-bad passage.
  5. Inspect the top results for a sample of real queries and classify misses as vocabulary, granularity or ranking failures.
  6. Compare against a lexical baseline on the same judged set before replacing anything.

Trade-offs

ChoiceGood forCostFails when
Dense-only retrievalParaphrase and cross-lingual matchingMisses exact identifiers and rare termsQueries contain codes, names or numeric constraints
Low similarity floorHigh recallIrrelevant passages reach the generatorThe corpus is heterogeneous and queries are narrow
High similarity floorFewer distractors and an abstain pathLegitimate paraphrases get filtered outThe embedding model is weak on the domain
Large chunksEnough context per hitThe vector averages several topicsAnswers live in one sentence of a long section
Trade-off mapInteractive

Good for

Paraphrase and cross-lingual matching

Cost

Misses exact identifiers and rare terms

Fails when

Queries contain codes, names or numeric constraints

  1. Queries contain codes, names or numeric constraints
  2. The corpus is heterogeneous and queries are narrow
  3. The embedding model is weak on the domain
  4. Answers live in one sentence of a long section

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Dense-only retrieval is good for Paraphrase and cross-lingual matching, costs Misses exact identifiers and rare terms, and fails when Queries contain codes, names or numeric constraints. Low similarity floor is good for High recall, costs Irrelevant passages reach the generator, and fails when The corpus is heterogeneous and queries are narrow. High similarity floor is good for Fewer distractors and an abstain path, costs Legitimate paraphrases get filtered out, and fails when The embedding model is weak on the domain. Large chunks is good for Enough context per hit, costs The vector averages several topics, and fails when Answers live in one sentence of a long section

Failure modes

  • Threshold copied from another corpus. Either everything passes or nothing does, and the team blames the model. Debug: plot the score distribution of labelled relevant and irrelevant pairs and pick a floor from the overlap.
  • Topic match mistaken for answer. The retrieved passage is about the right product but the wrong plan tier. Debug: read the top three passages and check whether the distinguishing qualifier is present.
  • Query and document preprocessing diverge. Queries are truncated or lowercased differently from documents, so distances shift. Debug: embed one known query-document pair through both paths and compare the resulting similarity.

In production

Monitor the distribution of top-1 similarity and the share of queries below the abstain floor. Re-embed and re-calibrate whenever the model version changes, and keep a lexical fallback for identifier queries.

Common misconceptions

People think: Semantic search understands the question.

Actually: It measures similarity between two texts; it has no notion of which text answers which question.

People think: A similarity score is a confidence.

Actually: The score is relative to the corpus and the model, so a 0.8 on one index can mean something entirely different on another.

Interview angle

The crux is the bi-encoder trade-off: documents are embedded without the query, so ranking is topical rather than answer-level. Discuss calibrating the similarity floor and the identifier failure that pushes teams to hybrid search. Saying the model understands meaning, without explaining why an exact part number is missed, is the usual weak answer.

Sources