Why it matters
Exact match fails a correct answer that uses different words, so teams reach for similarity scores that compare meaning. Those scores solve paraphrase and introduce a new problem: two sentences about the same topic sit close together even when one contradicts the other. The metric is useful when its failure mode is understood, and dangerous when a similarity threshold is treated as a correctness threshold.
Mental model
It is two summaries landing in the same neighbourhood. The metaphor breaks because a lie about the same subject lives on the same street as the truth, and distance in embedding space cannot tell you which house is which at all.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Modelled score distributions for correct paraphrases, contradictions and unrelated pairs over a 0–1 similarity score, each curve normalised to its own class, with a cut-off you can move. The distributions are illustrative; the overlap between the correct and contradiction curves is the claim.
What to notice. At a 0.75 cut-off, 82.5% of correct answers clear it and 28.4% of contradictions clear it too. Raise the cut-off to 0.90 and only 14.3% of correct answers survive while 0.3% of contradictions still pass, so no single threshold separates truth from topical closeness. The score ranks candidates; it does not decide a claim.
Text version
Three class-normalised distributions over a similarity score from 0 to 1. Correct paraphrases peak at 0.82 and 10.6% of that class per bin; contradictions peak at 0.71 and 11.3%; unrelated pairs sit near 0.34 and reach 8.0%. Because each curve is scaled to its own class, the area above the cut-off under a curve is the share of that class the panel reports. A vertical cut-off line at 0.75 divides the plot, and a panel reports 82.5% of correct answers, 28.4% of contradictions and under 0.1% of unrelated pairs. A slider moves the cut-off from 0.50 to 0.90 and all three shares update.
- Decide whether the task needs similarity to a reference or agreement with a fact, since those require different metrics.
- Pick an embedding model and record its version beside every score.
- Calibrate the threshold on labelled pairs that include correct paraphrases and similar-sounding contradictions.
- Report the score distribution and inspect the tails where the metric disagrees with human labels.
- For factual claims, decompose the answer and check each claim against evidence instead of scoring the whole text.
- Re-calibrate whenever the embedding model or the answer style changes.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Embedding similarity to a reference | Cheap, tolerant of paraphrase | Rewards topical overlap, misses contradiction | Answer must be judged for truth |
| Model-graded semantic match | Handles nuance | Judge cost and bias | Scale is large and budget fixed |
| Claim decomposition | Fact-level signal | Pipeline complexity | Answers are short and opinion-based |
| N-gram overlap | Trivial and fast | Punishes paraphrase | Wording varies legitimately |
Good for
Cheap, tolerant of paraphrase
Cost
Rewards topical overlap, misses contradiction
Fails when
Answer must be judged for truth
- Answer must be judged for truth
- Scale is large and budget fixed
- Answers are short and opinion-based
- Wording varies legitimately
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Embedding similarity to a reference is good for Cheap, tolerant of paraphrase, costs Rewards topical overlap, misses contradiction, and fails when Answer must be judged for truth. Model-graded semantic match is good for Handles nuance, costs Judge cost and bias, and fails when Scale is large and budget fixed. Claim decomposition is good for Fact-level signal, costs Pipeline complexity, and fails when Answers are short and opinion-based. N-gram overlap is good for Trivial and fast, costs Punishes paraphrase, and fails when Wording varies legitimately
Failure modes
- Similarity inversion. A contradicting answer scores higher than a correct paraphrase. Debug: inspect labelled pairs where the score order disagrees with the human order.
- Threshold transplant. A cutoff tuned on one corpus fails on another. Debug: recompute the precision-recall curve on your own labelled pairs.
- Reference bias. The reference is wrong or incomplete and every answer is scored against it. Debug: have a second annotator audit a sample of references.
In production
Track score distribution drift and disagreement with sampled human labels. Pin the embedding model version used for scoring so historical numbers stay comparable.
Common misconceptions
People think: High semantic similarity means the answer is correct.
Actually: It means the answer is about the same thing, and contradictions can score high.
People think: Semantic metrics remove the need for references.
Actually: Reference-free scoring replaces the reference with a model's judgement and inherits its biases.
Interview angle
Separate topical similarity from factual agreement and calibrate on labelled pairs. A strong answer names the inversion failure and says when a claim-level check is required. Proposing cosine similarity as a quality score, with no threshold or reference audit, is the weak response.
Sources
- Zhang et al., BERTScore — token-level embedding similarity for generated text.
- Liu et al., G-Eval — model-based scoring with explicit criteria.
- Zhao et al., MoverScore — soft alignment between generated and reference text.