Why it matters
String matching fails on the first paraphrase. A user asks "how do I get my money back" and the policy document says "refund eligibility". A keyword index sees no shared term and returns nothing, while a reader sees an obvious match. Embeddings exist to make that match computable: both texts become vectors, and the vectors are close.
That single trick powers more than search. Deduplication, clustering, topic labelling, recommendation, few-shot example selection and evaluation all use the same primitive. In a RAG system the embedding model is the component that decides what "relevant" means before any reranker or generator runs, so a mismatch between the model's notion of similarity and the product's notion of relevance is a systemic problem, not a tuning issue.
Mental model
An embedding is a coordinate on a very high-dimensional map drawn by a model that was trained to put paraphrases in the same neighbourhood. "Refund" and "money back" end up near each other because training pairs taught the map that they co-occur in meaning.
Three things break the map metaphor. First, the map has hundreds or thousands of dimensions and none of them is labelled — you cannot read off "this axis is formality". Second, the map is drawn for one purpose: a model trained on general web text will place two product SKUs close because both are products, which is exactly the distinction a support search needs. Third, the map is not stable across models — coordinates from two embedding models are unrelated even if the dimensions match.
How it works
querytop-k neighbour (cosine)wordedge leaves the clusterclusters: royalty · animals · vehicles · finance · food
Twenty-four synthetic words in five clusters. Pick a query from the plot or the select, then step to see which words the cosine score ranks closest.
synthetic score · 2 dp
Step to neighbours to rank the closest words by cosine.
Illustrative data: 24 words, a hand-placed 2-D layout and a synthetic seven-dimensional vector per word. Scores come from those vectors, not from a real embedding model, so read the order, not the absolute value.
Figure. Twenty-four synthetic words in five clusters of four or five. Each word carries a seven-dimensional vector; the plot shows its first two coordinates, hand-placed, not a PCA or t-SNE projection. The ranked scores are cosine similarity over all seven dimensions, computed from those synthetic vectors.
What to notice. Retrieval uses the order of the neighbour list, not the absolute position and not the raw score. A five-word cluster keeps k ≤ 4 inside itself; the four-word food cluster crosses at k = 4, when loan joins apple's three food neighbours. In general the first cross-cluster neighbour appears at k equal to the query cluster's word count.
Source: Synthetic vocabulary and vectors written for this figure; no embedding model was run. Clusters and positions are hand-placed for legibility.
Text version
A scatter of twenty-four labelled synthetic words in five clusters — royalty, animals, vehicles, finance and food — arranged around an empty origin. The default frame shows query loan in the finance cluster with k = 5: its four finance neighbours stock, bond, ledger and tax are ringed and joined to it in teal, and the fifth, apple from the food cluster at synthetic cosine 0.43, is ringed with a dashed edge that marks an edge leaving the cluster. The rank panel lists the five from 0.99 down to 0.43. Clicking a word or choosing it in the select moves the query, and a k slider sets the neighbourhood from three to eight. Hovering or focusing a word reports its label, cluster, synthetic cosine to two decimals and rank. A step control rewinds to the bare points and then to the unmarked query.
- Tokenise. Text is split into subword tokens and truncated or pooled to the model's maximum input length. Anything beyond that limit is silently dropped, so long documents must be chunked first.
- Encode. A transformer reads the tokens with full attention, so each token's representation is conditioned on the whole input. This is why the same word means different things in different sentences.
- Pool. A single vector is extracted, usually by mean-pooling token states or taking a designated summary token. The pooling method is part of the model's contract, not a free choice.
- Normalise. Many pipelines scale to unit length so that dot product equals cosine similarity, which makes inner-product indexes usable for cosine search.
- Store and index. Vectors go into a vector database with their text and metadata; an approximate nearest-neighbour index makes search sublinear.
- Embed the query at request time. The same model, with the same instruction prefix if the model expects one, embeds the question. Asymmetry matters: some models are trained for short-query-to-long-passage retrieval and degrade when used symmetric-style.
Dense retrieval models are usually trained with a contrastive objective: pull a query toward its relevant passage and push it away from in-batch negatives. For a query , passage and negatives :
The temperature controls how sharply the model separates near and far pairs. Hard negatives — passages that look relevant but are not — do most of the work in making the space useful for retrieval rather than mere topical clustering.
A minimal implementation
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
def embed(texts: list[str], batch_size: int = 64) -> list[list[float]]:
vectors = model.encode(
texts,
batch_size=batch_size,
normalize_embeddings=True, # unit length: dot product == cosine
convert_to_numpy=True,
)
return vectors.tolist()
# Index time and query time must use the same model, prefix and normalisation.
docs = embed(["Refunds are available within 14 days of purchase."])
query = embed(["can I get my money back"])[0]
score = sum(a * b for a, b in zip(query, docs[0]))Record the model identifier and a hash of its configuration next to the index. Re-embedding a corpus because of a silent model update is one of the more expensive avoidable incidents in this space.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Small general model | Fast baseline, high throughput | Weaker on jargon and fine distinctions | Domain vocabulary differs from web text |
| Large general model | Broad semantic coverage | Higher latency and storage per vector | Queries hinge on identifiers, codes or exact phrases |
| Domain fine-tuned model | In-house vocabulary and task shape | Labelled pairs, training, re-embedding | The labelled set is small or unrepresentative |
| Truncated dimensions (Matryoshka) | Cheaper storage and faster search | Some accuracy loss at aggressive cuts | Recall is already marginal at full dimension |
| Quantised vectors | Large corpora on modest hardware | Approximate distances need rescoring | Scores feed a threshold rather than a ranking |
Good for
Fast baseline, high throughput
Cost
Weaker on jargon and fine distinctions
Fails when
Domain vocabulary differs from web text
- Domain vocabulary differs from web text
- Queries hinge on identifiers, codes or exact phrases
- The labelled set is small or unrepresentative
- Recall is already marginal at full dimension
- Scores feed a threshold rather than a ranking
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Small general model is good for Fast baseline, high throughput, costs Weaker on jargon and fine distinctions, and fails when Domain vocabulary differs from web text. Large general model is good for Broad semantic coverage, costs Higher latency and storage per vector, and fails when Queries hinge on identifiers, codes or exact phrases. Domain fine-tuned model is good for In-house vocabulary and task shape, costs Labelled pairs, training, re-embedding, and fails when The labelled set is small or unrepresentative. Truncated dimensions (Matryoshka) is good for Cheaper storage and faster search, costs Some accuracy loss at aggressive cuts, and fails when Recall is already marginal at full dimension. Quantised vectors is good for Large corpora on modest hardware, costs Approximate distances need rescoring, and fails when Scores feed a threshold rather than a ranking
Failure modes
- Domain mismatch. "ERR_2041" and its explanation are not neighbours in a general model, so exact-identifier queries fail. Debug: take 30 failed queries, check whether the correct passage appears at all in the top 100, then test the same queries with a lexical index — if BM25 wins, you need hybrid search or a domain model.
- Version drift. Half the corpus is embedded with an older model after an upgrade, and distances become meaningless. Debug: store the model name per vector and assert one distinct value per collection; a mixed collection shows up as an unexplained recall drop.
- Silent truncation. Long chunks are cut at the token limit and the discriminating sentence is dropped. Debug: compare token counts per chunk against the model limit and log how many inputs exceed it.
- Threshold misuse. A fixed cut-off like 0.75 is ported from one model to another and either blocks everything or passes everything. Debug: plot score distributions for known-relevant and known-irrelevant pairs on the current model; calibrate per model and per query type.
In production
Embedding cost is dominated by ingestion, not query time — queries are still embedded on the request path, but once per query rather than once per chunk. The operational focus is ingestion throughput and re-indexing rather than per-request price. Batch aggressively, cache by content hash so unchanged documents are never re-embedded, and treat model upgrades as a migration: build a second index, evaluate both on a labelled set, then flip.
Watch three numbers: recall@k on a labelled query set, the share of queries whose top score is below the model's useful range, and the age of the oldest vector in the index. The third catches connector failures before users do.
Common misconceptions
People think: Embeddings capture meaning in general.
Actually: They capture the distinctions the training objective rewarded. A model trained for topical similarity often ranks a competing product above the correct clause.
People think: A similarity score is a confidence you can threshold.
Actually: Cosine similarities are not calibrated across models or queries. The same 0.8 can mean 'obvious match' for one query and 'weakest candidate' for another.
Interview angle
Expect to be asked why a semantically obvious passage was not retrieved. The answer interviewers want separates the three layers: the model's notion of similarity, the chunk's content, and the index's search parameters. Mentioning that query and document embeddings must come from the same model — and that upgrading one requires re-embedding everything — signals production experience. See embedding interview questions.
Sources
- Reimers & Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — the bi-encoder formulation behind most retrieval embeddings.
- Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering — contrastive training with in-batch negatives for retrieval.
- Muennighoff et al., MTEB: Massive Text Embedding Benchmark — why embedding quality is task-dependent and must be measured per task.
- Kusupati et al., Matryoshka Representation Learning — nested representations that allow truncating dimensions.