Why it matters
Models produce citations that look right: a real passage id, a real-looking quote, a claim the passage does not make. Users trust the marker and stop reading. Validation exists to break that trust loop by testing support mechanically, which requires sentence-level decomposition and a check that is strict enough to catch a paraphrase that reverses the meaning. Without that check, a citation is a pointer nobody has followed.
Mental model
Citation validation is a fact-checker with the source open beside the draft, matching each sentence to a line. The metaphor breaks because a fact-checker understands implication, while a lexical matcher only sees shared words and a weak entailment model can be fooled by negation.
How it works
Support is a relation between a claim and a span: a pointer can exist, and the span can still fail to entail the claim.
Step 1 of 6
Read the shape
Three claims on the left, three passages on the right. Citation edges point from claim to span; the return edge is support; the conflict edges are the two ways a real citation fails.
Narrow layout: nodes in reading order, each followed by its outgoing edges. The wide drawing shows the same graph as geometry.
claim
3 connections
Claim A
refund within 14 days
A claim whose cited span exists verbatim and entails it. The pointer and the support relation agree, which is the success case the validation is built to distinguish from the others.
No outgoing edge.
document · claim · 7 nodes · 9 edges · fixed authored layout, no simulation · list
Figure. Validation tests two relations, not one. A citation edge points from a claim to a character span, and a support edge runs back from the span only when the passage entails the claim. Claim A has both; Claim B has a real quote that contradicts it; Claim C points at a chunk that was dropped before the prompt. The conflict edges are where existence and support come apart.
What to notice. The citation edge is cheap and checkable, the return edge is a judgement over the claim and the span, and the conflict edges are the failures a marker alone hides. The figure omits the validator's own model and error rate, the sampling policy and the repair prompt. The right-document, wrong-passage case is not shown. Spans, offsets and claim text are illustrative; it shows one answer, not a benchmark.
Text version
Seven nodes. An answer splits into claims A, B and C. Claim A cites passage A at span 412 to 468, and a return edge says the passage entails it. Claim B cites passage B at span 88 to 121, and a conflict edge from passage B says the passage contradicts the claim because the feature is not included. Claim C cites passage C, which was dropped at assembly, and a conflict parallel edge says it is not in the prompt. Steps light the whole graph, the claim split, span resolution and entailment, the negation flip, the unseen chunk, and repair or abstention.
- Split the answer into atomic claims, since one sentence can contain a supported fact and an unsupported inference.
- Resolve each claim's citation marker to the exact passage and character span that was in the prompt.
- Run a cheap check first: does the quoted text appear verbatim in the cited passage?
- Run a support check on the claim and the cited span, not the whole document, using entailment or a cross-encoder judgement.
- Classify failures as missing, wrong-source or contradicted, because each has a different repair.
- Repair by regenerating with the failing claim highlighted, or drop the claim and abstain; never add a citation the checker did not verify.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Verbatim quote matching | Deterministic and fast | Misses paraphrase and misreading | Answers synthesise rather than quote |
| Entailment model | Catches paraphrase and negation | Model cost and its own error rate | The domain language is far from training data |
| Validate every sentence | Highest assurance | Latency and cost per answer | Answers are long and interactive |
| Validate only cited sentences | Cheap | Uncited claims pass unchecked | The model omits citations when unsure |
Good for
Deterministic and fast
Cost
Misses paraphrase and misreading
Fails when
Answers synthesise rather than quote
- Answers synthesise rather than quote
- The domain language is far from training data
- Answers are long and interactive
- The model omits citations when unsure
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Verbatim quote matching is good for Deterministic and fast, costs Misses paraphrase and misreading, and fails when Answers synthesise rather than quote. Entailment model is good for Catches paraphrase and negation, costs Model cost and its own error rate, and fails when The domain language is far from training data. Validate every sentence is good for Highest assurance, costs Latency and cost per answer, and fails when Answers are long and interactive. Validate only cited sentences is good for Cheap, costs Uncited claims pass unchecked, and fails when The model omits citations when unsure
Failure modes
- Citation to the right document, wrong passage. The id resolves but the supporting sentence is elsewhere, so a reader checking the link finds nothing. Debug: for a sample, open each cited span and confirm the claim's key terms appear in it.
- Negation flip. The passage says a feature is not included and the answer says it is, with a citation that shares all the words. Debug: test the checker on negated pairs; a purely lexical check will pass both.
- Validator agrees with the generator. The same model both writes and validates, so its misreading is confirmed. Debug: validate with a different model or a human sample and compare disagreement rates.
In production
Report support rate per answer and per source document. Alert when a document's support rate falls, which often means it was re-parsed and its spans shifted.
Common misconceptions
People think: A citation is evidence of grounding.
Actually: It is a pointer; support is a separate property that has to be tested.
People think: Validation should always trigger a rewrite.
Actually: A rewrite can produce a new unsupported claim; dropping the claim is often the safer repair.
Interview angle
Distinguish attribution from support, validate at the claim level, and name negation as the case lexical matching misses. Describe a repair policy that can abstain. Asking the model to cite sources and calling that verification is the weak answer.
Sources
- Gao et al., Enabling Large Language Models to Generate Text with Citations — automatic evaluation of whether generated citations are supported.
- Rashkin et al., Measuring Attribution in Natural Language Generation Models — attribution as a separately measurable property of generated text.
- Es et al., RAGAS — reference-free metrics for faithfulness of retrieved-augmented answers.