Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 10 of 57

Similarity metrics

0 of 57 complete

Why it matters

An embedding is only useful if you can order pairs by closeness, and the ordering depends entirely on the metric. Choose cosine and you compare angles; choose inner product and a long vector can outrank a better-aligned short one; choose Euclidean and the geometry changes again. These are not cosmetic differences — each one defines a different neighbour set, and therefore a different retrieval result.

The choice is also constrained by infrastructure. Approximate indexes are built for a specific metric, and switching later means rebuilding the index. Most real incidents here are not subtle modelling problems; they are an index configured for inner product, a query path that forgot to normalise, and a recall drop nobody could explain.

Mental model

Think of embeddings as points and the metric as the ruler. Cosine measures the angle between two arrows drawn from the origin, so it ignores how long the arrows are. Dot product multiplies lengths as well as alignment, so it rewards confidence or frequency as well as direction. Euclidean is the ordinary straight-line distance you would measure with a physical ruler.

The ruler metaphor breaks because there is no physical space to check against. A "short distance" in embedding space has no units and no meaning outside the model that produced it. Two passages 0.1 apart might be near-duplicates or merely both about refunds, and only labelled data can tell you which.

How it works

Three metrics · one candidate setInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. One query and six synthetic candidates in a hand-placed 2-D layout. Scores are computed from the plotted coordinates: cosine and dot are similarities, where higher ranks first; euclidean is a distance, where lower ranks first.

What to notice. The order is the metric. Cosine puts reset password link first at 1.00 and the long request second at 0.99. Dot product promotes the long request to first (0.55 against 0.51) because its vector is longer. Euclidean puts account locked out first at 0.28. Compared with cosine's top two, euclidean's top two are account locked out (0.28) and reset password link (0.50); the long request falls to fourth (0.59), so it is out at k = 2 as well. At k = 3 change email address (0.52) displaces it too. Directions and lengths are hand-placed so the flips are visible; a real embedding would not separate this neatly.

Text version

A scatter of six labelled synthetic candidates around a query, reset password, in a hand-placed 2-D layout. The default frame uses cosine and k = 3: reset password link at 1.00, the long reset request at 0.99 and account locked out at 0.97 are ringed and joined to the query in teal, and the rank panel lists them in that order, followed by change email address at 0.66 and two off-topic candidates with negative scores. A metric control switches the same points to dot product, which moves the long reset request to first, and to euclidean distance, which moves account locked out to first at 0.28 and brings change email address into the third slot at 0.52. A slider sets k from one to five. The caption states that the coordinates are hand-placed and the scores synthetic.

  1. Fix the embedding contract. Whatever normalisation the model's documentation specifies is part of the pipeline. Many retrieval models are trained for cosine or inner product on normalised vectors; changing that silently changes what the model was tuned to do.
  2. Normalise consistently. If you normalise, do it at ingestion and at query time, and persist the normalised vector — not the raw one. Storing raw vectors and normalising at query time guarantees a mismatch.
  3. Choose the index metric to match. Vector databases expose the metric per collection and use it to build the index. Cosine and inner product on unit vectors are interchangeable; on unnormalised vectors they are not.
  4. Rank, then interpret. Use the score to order candidates. Do not treat it as a probability, and do not compare scores across queries or models.
  5. Calibrate any threshold. If you need an abstain path or a filter, derive the cut-off from labelled pairs on the current model, and re-derive after any model change.
  6. Validate with retrieval metrics. Recall@k and nDCG@k on a labelled set are the ground truth for "is this metric working", not the visual plausibility of the top result.

Cosine similarity is the normalised dot product:

cos⁡(a,b)=a⋅b∥a∥ ∥b∥\cos(a, b) = \frac{a \cdot b}{\lVert a \rVert \, \lVert b \rVert}

For unit vectors, Euclidean distance is a monotone function of cosine, which is why the two rank identically:

∥a−b∥2=∥a∥2+∥b∥2−2a⋅b=2−2cos⁡(a,b)\lVert a - b \rVert^2 = \lVert a \rVert^2 + \lVert b \rVert^2 - 2 a \cdot b = 2 - 2\cos(a,b)

That equivalence is exactly why the normalisation bug is so common: on a small test set with short texts the rankings happen to agree, so the mismatch ships.

A minimal implementation

import numpy as np
 
def cosine(a: np.ndarray, b: np.ndarray) -> float:
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
 
def l2(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.linalg.norm(a - b))
 
# A long document vector has a larger norm than a short query vector.
doc = np.array([0.9, 0.1, 0.0])
query = np.array([0.6, 0.8, 0.0])
verbose_doc = doc * 4.0
 
print(cosine(query, doc))          # 0.68
print(cosine(query, verbose_doc))  # 0.68  — direction only
print(query @ doc, query @ verbose_doc)  # 0.62 vs 2.48 — magnitude matters

The dot-product column shows the trap: an unnormalised inner-product index ranks the verbose duplicate first, purely because its vector is longer. If the corpus was built without normalisation, that is a systematic bias toward long or repetitive documents.

Trade-offs

MetricGood forCostFails when
CosineText retrieval with models trained on normalised embeddingsRequires consistent normalisationVector magnitude carries real signal (frequency, confidence)
Inner productRecommendation and models trained for maximum inner product search (MIPS)Magnitude bias on unnormalised dataDocuments vary a lot in length or salience
Euclidean (L2)Clustering, anomaly detection, image featuresSensitive to unnormalised scaleVectors are unnormalised text embeddings
Reranker scoreOrdering a small shortlist preciselyOne forward pass per candidateYou need an absolute relevance threshold across queries
Reciprocal rank fusionCombining lists with incomparable scoresIgnores score magnitude by designOne list is much stronger and should dominate
Trade-off mapInteractive

Good for

Text retrieval with models trained on normalised embeddings

Cost

Requires consistent normalisation

Fails when

Vector magnitude carries real signal (frequency, confidence)

  1. Vector magnitude carries real signal (frequency, confidence)
  2. Documents vary a lot in length or salience
  3. Vectors are unnormalised text embeddings
  4. You need an absolute relevance threshold across queries
  5. One list is much stronger and should dominate

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Cosine is good for Text retrieval with models trained on normalised embeddings, costs Requires consistent normalisation, and fails when Vector magnitude carries real signal (frequency, confidence). Inner product is good for Recommendation and models trained for maximum inner product search (MIPS), costs Magnitude bias on unnormalised data, and fails when Documents vary a lot in length or salience. Euclidean (L2) is good for Clustering, anomaly detection, image features, costs Sensitive to unnormalised scale, and fails when Vectors are unnormalised text embeddings. Reranker score is good for Ordering a small shortlist precisely, costs One forward pass per candidate, and fails when You need an absolute relevance threshold across queries. Reciprocal rank fusion is good for Combining lists with incomparable scores, costs Ignores score magnitude by design, and fails when One list is much stronger and should dominate

Failure modes

  • Metric mismatch between index and query. The collection is built for cosine, the application sends unnormalised vectors, and neighbours drift. Debug: take a known query, compute the top 10 by brute force with the intended metric, and compare against the index's top 10. Divergence localises the bug immediately.
  • Threshold portability. A cut-off tuned on one model blocks every result on another. Debug: plot score distributions for labelled relevant and irrelevant pairs on the current model; if the distributions overlap, no single threshold works and you need a reranker or an abstain classifier.
  • Ties from near-duplicate chunks. Five copies of the same sentence fill the top of the list and crowd out the answer. Debug: count distinct source documents in the top kk; if the count is far below kk, add deduplication or maximal marginal relevance.
  • Score fusion without normalisation. Dense scores in [0,1][0,1] are added to BM25 scores in the tens, and lexical dominates invisibly. Debug: log the contribution of each retriever per query; if one list always wins, fuse ranks rather than raw scores.

In production

Normalisation is a schema decision, not a runtime one: store unit-length vectors if the model expects cosine, and assert it on write. Some databases return a distance and others a similarity, so wrap the conversion in one function rather than sprinkling signs through the code — an inverted comparison is easy to ship and hard to notice, because results still look roughly plausible.

Watch recall@k, the score distribution of the top result per query, and the number of distinct documents per result set. The third catches duplicate-heavy corpora that metrics alone will not flag.

Common misconceptions

People think: Cosine and dot product are interchangeable.

Actually: They rank identically only when every vector has unit length. On unnormalised embeddings, dot product quietly rewards longer documents.

People think: A similarity of 0.8 means 80% relevant.

Actually: Similarity is an ordinal score from one model. Its scale differs across models, query lengths and domains, so it cannot be read as a probability.

Interview angle

The question is usually framed as "why did adding a second embedding model break recall?" A strong answer starts from the metric contract: which metric the index was built for, whether vectors were normalised at both ends, and whether scores from two models are comparable at all. Then it separates ranking quality from threshold quality — the first needs recall@k, the second needs labelled score distributions. Browse RAG interview questions for variants on this.

Sources