Why it matters
A retrieval system's central promise is that the answer comes from the documents. The characteristic failure is an answer that reads fluently, cites a real document, and contains a sentence the document never says. Retrieval-augmented generation reduces hallucination; it does not eliminate it, because the generator still writes prose and can blend parametric memory with retrieved evidence.
Faithfulness makes that failure measurable. It is the property you check when the question is "did the model stick to the evidence?", as distinct from "is the answer correct?" and "does it answer the question?". Splitting those three apart is what lets you route a failure to the retriever, the context assembler, or the generator.
Mental model
A copy editor with a source packet and no internet. For every sentence the writer submits, the editor checks whether the packet supports it, contradicts it, or says nothing about it. The editor does not judge whether the packet is right, and does not reward the writer for omitting difficult facts.
That last clause is where the analogy matters. An editor who deletes every uncertain sentence produces a perfectly supported, useless article. Faithfulness has the same loophole: "I could not find that in the documents" is fully faithful. Always read the score next to how often the system actually answers.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. The share of answers whose supported-claim share reaches each threshold, so the curve is read right to left. A mean of 83.5% is compatible with a tail of answers that are mostly unsupported. The three-block model is stated in the footnote.
What to notice. The mean faithfulness is 83.5% and 62% of answers have every claim supported. The curve shows what the mean hides: 14% of answers sit below half their claims supported and 29% fall below 0.80. A dashboard reading only the mean would pass the tail, which is the population a reviewer needs to read.
Text version
Share of answers at or above each supported-claim threshold, from 0 to 1. The curve starts at 100% at zero, stays near 97% at 0.10, and falls to 86% at 0.50, 71% at 0.80 and 62% at 1.00, where every claim must be supported. A marker below the right end reads 62% supported; a second marker just right of 0.80 reads 29%; a vertical line marks half the claims supported. The stat panel spells out that 29% is the share below 0.80 and 62% the fully supported share. Two steps introduce the score and then the tail; a panel reports the mean faithfulness, the fully supported share and the share below half.
- Decompose the answer into atomic claims. One fact per claim, with pronouns resolved. "The Pro plan costs 40 and includes SSO" is two claims, and they can have different verdicts.
- Search only the supplied context. The verifier gets the answer and the exact context the generator saw. Do not let it consult the open web or its own knowledge; that measures truth, not grounding.
- Classify each claim. Supported (entailed by a span), contradicted (a span says otherwise), or unsupported (no span addresses it). Keeping contradiction separate matters: a contradicted claim is a worse failure than an unsupported one.
- Score the answer. Faithfulness is the supported share of claims. Report the contradiction rate separately and keep the unsupported claims as text, because they are the debugging artefact.
- Aggregate with a denominator you trust. Average per answer, but also report the share of answers with at least one unsupported claim. The mean hides the case where 10% of answers are badly wrong.
- Calibrate the verifier. Hand-label a sample of claims and measure agreement. Claim decomposition and entailment are both model judgements, and both can be lenient.
- Close the loop. Unsupported claims usually trace to one of three causes: the fact was never retrieved; it was retrieved but truncated out of the prompt; or the generator added it anyway. Each has a different fix.
A claim-level check
from typing import Callable, Literal
Verdict = Literal["supported", "contradicted", "unsupported"]
def faithfulness(
answer: str,
context: str,
split_claims: Callable[[str], list[str]],
entail: Callable[[str, str], Verdict],
) -> dict | None:
claims = split_claims(answer)
if not claims:
return None # abstention: not evidence of grounding
verdicts = [entail(claim, context) for claim in claims]
supported = sum(v == "supported" for v in verdicts)
contradicted = sum(v == "contradicted" for v in verdicts)
return {
"score": supported / len(claims),
"contradicted_rate": contradicted / len(claims),
"unsupported_claims": [c for c, v in zip(claims, verdicts) if v != "supported"],
}Returning None for an empty answer is deliberate: a system that refuses everything should not score 1.0. In the aggregate report, count those answers separately from faithful ones.
Trade-offs
| Method | Good for | Cost | Fails when |
|---|---|---|---|
| LLM claim judge | Nuanced support, multi-sentence claims | Token cost, judge bias | Claims are long or the context is contradictory |
| NLI entailment model | High-volume, low-latency checks | Needs a domain-appropriate model | Claims need arithmetic or world knowledge |
| Citation string match | Cheap sanity checks | Near-zero cost | The cited span is paraphrased or indirect |
| Human audit | Calibration and disputed cases | Slow and expensive | Sample is too small to cover segments |
| End-to-end judge | One number for a dashboard | Opaque when it disagrees | You need to know which claim failed |
Good for
Nuanced support, multi-sentence claims
Cost
Token cost, judge bias
Fails when
Claims are long or the context is contradictory
- Claims are long or the context is contradictory
- Claims need arithmetic or world knowledge
- The cited span is paraphrased or indirect
- Sample is too small to cover segments
- You need to know which claim failed
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: LLM claim judge is good for Nuanced support, multi-sentence claims, costs Token cost, judge bias, and fails when Claims are long or the context is contradictory. NLI entailment model is good for High-volume, low-latency checks, costs Needs a domain-appropriate model, and fails when Claims need arithmetic or world knowledge. Citation string match is good for Cheap sanity checks, costs Near-zero cost, and fails when The cited span is paraphrased or indirect. Human audit is good for Calibration and disputed cases, costs Slow and expensive, and fails when Sample is too small to cover segments. End-to-end judge is good for One number for a dashboard, costs Opaque when it disagrees, and fails when You need to know which claim failed
Failure modes
Abstention inflates the score. The system learns to say "the documents do not cover this" and faithfulness approaches 1.0 while usefulness collapses. Debug by reporting answer rate and coverage beside faithfulness, and by sampling refusals to check whether the evidence was actually present.
Decomposition merges claims. "The plan is available in the EU and costs less than the Pro tier" becomes one claim, and a verifier marks it supported because half of it is. Debug by printing the claim list for every failing answer and looking for conjunctions.
The verifier sees different context. Evidence was retrieved but cut during context assembly, so a true claim looks unsupported. Debug by logging the exact prompt context next to the verifier input and diffing them.
Lenient entailment. The verifier accepts a claim because the document discusses the same topic. Debug with an adversarial calibration set of near-miss pairs, and track agreement on the unsupported class rather than overall accuracy.
In production
Run faithfulness on a sample of live traffic and on every offline eval run. As an online guardrail, block or rewrite only contradicted claims above a confidence threshold; unsupported claims are better surfaced as "not found in sources" than silently removed. Store the claim, verdict, supporting span and context ID with each result, because that record is what makes a complaint investigable.
Cost is proportional to claims per answer, so long answers are expensive to verify. Cache by answer hash and verifier version, and prefer verifying the claims in the final answer rather than every draft. Track faithfulness by segment: a drop in one document collection usually means ingestion, not generation.
Common misconceptions
People think: Faithfulness and factual correctness are the same thing.
Actually: Faithfulness is support by the supplied context. A stale or wrong document yields a faithful, false answer. Factuality requires an external check.
People think: If the answer cites a source, it is grounded.
Actually: A citation can be decorative, mismatched, or attached to a claim the source does not support. Citation quality needs its own check.
People think: High faithfulness means the RAG system is working.
Actually: A system that refuses hard questions, or retrieves nothing and says so, can be perfectly faithful and useless. Read it with coverage and answer relevance.
Interview angle
"How would you detect hallucination in a RAG answer?" is the entry point; the follow-up is "how do you know your detector works?" A strong answer separates faithfulness from factuality, describes claim decomposition and per-claim verdicts, keeps contradiction distinct, calibrates the verifier against human labels, and pairs the score with coverage so abstention cannot game it. See the RAG bank and RAG evaluation.
Sources
- Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation — faithfulness as supported claims over total claims, with generated question sets.
- Min et al., FActScore: Fine-grained Atomic Evaluation of Factual Precision — atomic fact decomposition and support checking against a knowledge source.
- Gao et al., Enabling Large Language Models to Generate Text with Citations — citation correctness and completeness as separate measures.
- Tang et al., MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents — a smaller trained checker for document-grounded verification.
Related
- Citations — the interface faithfulness is measured against.
- RAG evaluation — retrieval and generation metrics together.
- Answer quality — correctness and relevance beyond grounding.
- Context metrics — whether the evidence arrived at all.
- Untrusted retrieval — when the context itself is hostile.