Skip to content
AI Engineer Guide.

concept · evaluation · intermediate · concept 13 of 28

Faithfulness

0 of 28 complete

Why it matters

A retrieval system's central promise is that the answer comes from the documents. The characteristic failure is an answer that reads fluently, cites a real document, and contains a sentence the document never says. Retrieval-augmented generation reduces hallucination; it does not eliminate it, because the generator still writes prose and can blend parametric memory with retrieved evidence.

Faithfulness makes that failure measurable. It is the property you check when the question is "did the model stick to the evidence?", as distinct from "is the answer correct?" and "does it answer the question?". Splitting those three apart is what lets you route a failure to the retriever, the context assembler, or the generator.

Mental model

A copy editor with a source packet and no internet. For every sentence the writer submits, the editor checks whether the packet supports it, contradicts it, or says nothing about it. The editor does not judge whether the packet is right, and does not reward the writer for omitting difficult facts.

That last clause is where the analogy matters. An editor who deletes every uncertain sentence produces a perfectly supported, useless article. Faithfulness has the same loophole: "I could not find that in the documents" is fully faithful. Always read the score next to how often the system actually answers.

How it works

The mean hides the tailInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. The share of answers whose supported-claim share reaches each threshold, so the curve is read right to left. A mean of 83.5% is compatible with a tail of answers that are mostly unsupported. The three-block model is stated in the footnote.

What to notice. The mean faithfulness is 83.5% and 62% of answers have every claim supported. The curve shows what the mean hides: 14% of answers sit below half their claims supported and 29% fall below 0.80. A dashboard reading only the mean would pass the tail, which is the population a reviewer needs to read.

Text version

Share of answers at or above each supported-claim threshold, from 0 to 1. The curve starts at 100% at zero, stays near 97% at 0.10, and falls to 86% at 0.50, 71% at 0.80 and 62% at 1.00, where every claim must be supported. A marker below the right end reads 62% supported; a second marker just right of 0.80 reads 29%; a vertical line marks half the claims supported. The stat panel spells out that 29% is the share below 0.80 and 62% the fully supported share. Two steps introduce the score and then the tail; a panel reports the mean faithfulness, the fully supported share and the share below half.

  1. Decompose the answer into atomic claims. One fact per claim, with pronouns resolved. "The Pro plan costs 40 and includes SSO" is two claims, and they can have different verdicts.
  2. Search only the supplied context. The verifier gets the answer and the exact context the generator saw. Do not let it consult the open web or its own knowledge; that measures truth, not grounding.
  3. Classify each claim. Supported (entailed by a span), contradicted (a span says otherwise), or unsupported (no span addresses it). Keeping contradiction separate matters: a contradicted claim is a worse failure than an unsupported one.
  4. Score the answer. Faithfulness is the supported share of claims. Report the contradiction rate separately and keep the unsupported claims as text, because they are the debugging artefact.
  5. Aggregate with a denominator you trust. Average per answer, but also report the share of answers with at least one unsupported claim. The mean hides the case where 10% of answers are badly wrong.
  6. Calibrate the verifier. Hand-label a sample of claims and measure agreement. Claim decomposition and entailment are both model judgements, and both can be lenient.
  7. Close the loop. Unsupported claims usually trace to one of three causes: the fact was never retrieved; it was retrieved but truncated out of the prompt; or the generator added it anyway. Each has a different fix.

A claim-level check

from typing import Callable, Literal
 
Verdict = Literal["supported", "contradicted", "unsupported"]
 
def faithfulness(
    answer: str,
    context: str,
    split_claims: Callable[[str], list[str]],
    entail: Callable[[str, str], Verdict],
) -> dict | None:
    claims = split_claims(answer)
    if not claims:
        return None  # abstention: not evidence of grounding
 
    verdicts = [entail(claim, context) for claim in claims]
    supported = sum(v == "supported" for v in verdicts)
    contradicted = sum(v == "contradicted" for v in verdicts)
    return {
        "score": supported / len(claims),
        "contradicted_rate": contradicted / len(claims),
        "unsupported_claims": [c for c, v in zip(claims, verdicts) if v != "supported"],
    }

Returning None for an empty answer is deliberate: a system that refuses everything should not score 1.0. In the aggregate report, count those answers separately from faithful ones.

Trade-offs

MethodGood forCostFails when
LLM claim judgeNuanced support, multi-sentence claimsToken cost, judge biasClaims are long or the context is contradictory
NLI entailment modelHigh-volume, low-latency checksNeeds a domain-appropriate modelClaims need arithmetic or world knowledge
Citation string matchCheap sanity checksNear-zero costThe cited span is paraphrased or indirect
Human auditCalibration and disputed casesSlow and expensiveSample is too small to cover segments
End-to-end judgeOne number for a dashboardOpaque when it disagreesYou need to know which claim failed
Trade-off mapInteractive

Good for

Nuanced support, multi-sentence claims

Cost

Token cost, judge bias

Fails when

Claims are long or the context is contradictory

  1. Claims are long or the context is contradictory
  2. Claims need arithmetic or world knowledge
  3. The cited span is paraphrased or indirect
  4. Sample is too small to cover segments
  5. You need to know which claim failed

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: LLM claim judge is good for Nuanced support, multi-sentence claims, costs Token cost, judge bias, and fails when Claims are long or the context is contradictory. NLI entailment model is good for High-volume, low-latency checks, costs Needs a domain-appropriate model, and fails when Claims need arithmetic or world knowledge. Citation string match is good for Cheap sanity checks, costs Near-zero cost, and fails when The cited span is paraphrased or indirect. Human audit is good for Calibration and disputed cases, costs Slow and expensive, and fails when Sample is too small to cover segments. End-to-end judge is good for One number for a dashboard, costs Opaque when it disagrees, and fails when You need to know which claim failed

Failure modes

Abstention inflates the score. The system learns to say "the documents do not cover this" and faithfulness approaches 1.0 while usefulness collapses. Debug by reporting answer rate and coverage beside faithfulness, and by sampling refusals to check whether the evidence was actually present.

Decomposition merges claims. "The plan is available in the EU and costs less than the Pro tier" becomes one claim, and a verifier marks it supported because half of it is. Debug by printing the claim list for every failing answer and looking for conjunctions.

The verifier sees different context. Evidence was retrieved but cut during context assembly, so a true claim looks unsupported. Debug by logging the exact prompt context next to the verifier input and diffing them.

Lenient entailment. The verifier accepts a claim because the document discusses the same topic. Debug with an adversarial calibration set of near-miss pairs, and track agreement on the unsupported class rather than overall accuracy.

In production

Run faithfulness on a sample of live traffic and on every offline eval run. As an online guardrail, block or rewrite only contradicted claims above a confidence threshold; unsupported claims are better surfaced as "not found in sources" than silently removed. Store the claim, verdict, supporting span and context ID with each result, because that record is what makes a complaint investigable.

Cost is proportional to claims per answer, so long answers are expensive to verify. Cache by answer hash and verifier version, and prefer verifying the claims in the final answer rather than every draft. Track faithfulness by segment: a drop in one document collection usually means ingestion, not generation.

Common misconceptions

People think: Faithfulness and factual correctness are the same thing.

Actually: Faithfulness is support by the supplied context. A stale or wrong document yields a faithful, false answer. Factuality requires an external check.

People think: If the answer cites a source, it is grounded.

Actually: A citation can be decorative, mismatched, or attached to a claim the source does not support. Citation quality needs its own check.

People think: High faithfulness means the RAG system is working.

Actually: A system that refuses hard questions, or retrieves nothing and says so, can be perfectly faithful and useless. Read it with coverage and answer relevance.

Interview angle

"How would you detect hallucination in a RAG answer?" is the entry point; the follow-up is "how do you know your detector works?" A strong answer separates faithfulness from factuality, describes claim decomposition and per-claim verdicts, keeps contradiction distinct, calibrates the verifier against human labels, and pairs the score with coverage so abstention cannot game it. See the RAG bank and RAG evaluation.

Sources

  • Citations — the interface faithfulness is measured against.
  • RAG evaluation — retrieval and generation metrics together.
  • Answer quality — correctness and relevance beyond grounding.
  • Context metrics — whether the evidence arrived at all.
  • Untrusted retrieval — when the context itself is hostile.