Skip to content
AI Engineer Guide.

concept · evaluation · intermediate · concept 5 of 28

Semantic metrics

0 of 28 complete

Why it matters

Exact match fails a correct answer that uses different words, so teams reach for similarity scores that compare meaning. Those scores solve paraphrase and introduce a new problem: two sentences about the same topic sit close together even when one contradicts the other. The metric is useful when its failure mode is understood, and dangerous when a similarity threshold is treated as a correctness threshold.

Mental model

It is two summaries landing in the same neighbourhood. The metaphor breaks because a lie about the same subject lives on the same street as the truth, and distance in embedding space cannot tell you which house is which at all.

How it works

Similar scores, opposite meaningsInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Modelled score distributions for correct paraphrases, contradictions and unrelated pairs over a 0–1 similarity score, each curve normalised to its own class, with a cut-off you can move. The distributions are illustrative; the overlap between the correct and contradiction curves is the claim.

What to notice. At a 0.75 cut-off, 82.5% of correct answers clear it and 28.4% of contradictions clear it too. Raise the cut-off to 0.90 and only 14.3% of correct answers survive while 0.3% of contradictions still pass, so no single threshold separates truth from topical closeness. The score ranks candidates; it does not decide a claim.

Text version

Three class-normalised distributions over a similarity score from 0 to 1. Correct paraphrases peak at 0.82 and 10.6% of that class per bin; contradictions peak at 0.71 and 11.3%; unrelated pairs sit near 0.34 and reach 8.0%. Because each curve is scaled to its own class, the area above the cut-off under a curve is the share of that class the panel reports. A vertical cut-off line at 0.75 divides the plot, and a panel reports 82.5% of correct answers, 28.4% of contradictions and under 0.1% of unrelated pairs. A slider moves the cut-off from 0.50 to 0.90 and all three shares update.

  1. Decide whether the task needs similarity to a reference or agreement with a fact, since those require different metrics.
  2. Pick an embedding model and record its version beside every score.
  3. Calibrate the threshold on labelled pairs that include correct paraphrases and similar-sounding contradictions.
  4. Report the score distribution and inspect the tails where the metric disagrees with human labels.
  5. For factual claims, decompose the answer and check each claim against evidence instead of scoring the whole text.
  6. Re-calibrate whenever the embedding model or the answer style changes.

Trade-offs

ChoiceGood forCostFails when
Embedding similarity to a referenceCheap, tolerant of paraphraseRewards topical overlap, misses contradictionAnswer must be judged for truth
Model-graded semantic matchHandles nuanceJudge cost and biasScale is large and budget fixed
Claim decompositionFact-level signalPipeline complexityAnswers are short and opinion-based
N-gram overlapTrivial and fastPunishes paraphraseWording varies legitimately
Trade-off mapInteractive

Good for

Cheap, tolerant of paraphrase

Cost

Rewards topical overlap, misses contradiction

Fails when

Answer must be judged for truth

  1. Answer must be judged for truth
  2. Scale is large and budget fixed
  3. Answers are short and opinion-based
  4. Wording varies legitimately

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Embedding similarity to a reference is good for Cheap, tolerant of paraphrase, costs Rewards topical overlap, misses contradiction, and fails when Answer must be judged for truth. Model-graded semantic match is good for Handles nuance, costs Judge cost and bias, and fails when Scale is large and budget fixed. Claim decomposition is good for Fact-level signal, costs Pipeline complexity, and fails when Answers are short and opinion-based. N-gram overlap is good for Trivial and fast, costs Punishes paraphrase, and fails when Wording varies legitimately

Failure modes

  • Similarity inversion. A contradicting answer scores higher than a correct paraphrase. Debug: inspect labelled pairs where the score order disagrees with the human order.
  • Threshold transplant. A cutoff tuned on one corpus fails on another. Debug: recompute the precision-recall curve on your own labelled pairs.
  • Reference bias. The reference is wrong or incomplete and every answer is scored against it. Debug: have a second annotator audit a sample of references.

In production

Track score distribution drift and disagreement with sampled human labels. Pin the embedding model version used for scoring so historical numbers stay comparable.

Common misconceptions

People think: High semantic similarity means the answer is correct.

Actually: It means the answer is about the same thing, and contradictions can score high.

People think: Semantic metrics remove the need for references.

Actually: Reference-free scoring replaces the reference with a model's judgement and inherits its biases.

Interview angle

Separate topical similarity from factual agreement and calibrate on labelled pairs. A strong answer names the inversion failure and says when a claim-level check is required. Proposing cosine similarity as a quality score, with no threshold or reference audit, is the weak response.

Sources