seniorPracticecurated
Answers cite real documents that do not actually contain the claim. How do you detect and fix bad citations?
30-second answer
- Citation accuracy and faithfulness are different failures: the claim can be right while the marker points at a passage that never states it.
- Verify mechanically by parsing markers, checking each resolves to a chunk actually in the prompt, then testing whether the cited span supports the sentence.
- Common causes are markers drifting when context is truncated, near-duplicate policy versions being confused, and the model citing the top-ranked chunk out of habit.
- Fix with numbered chunk IDs, cite-by-ID instructions, post-hoc verification that strips invalid markers, and version metadata carried into the prompt.
- Claim-level verification adds a model call per sentence and flags correct paraphrases, so decide whether a failure strips the marker or forces regeneration and measure that false-positive rate.
Likely follow-ups
Open a question to rehearse the answer.
When verification finds an unsupported citation, do you strip the marker or regenerate the answer?
- Strip when the claim is peripheral and the rest of the answer stands, so the marker goes and the sentence is removed or shown as unverified.
- Regenerate when the claim is central and the evidence was in the prompt, passing the failing claim and its cited span back so the repair is targeted.
- Never add a citation the checker did not verify, because a repair that invents support is worse than an abstention.
How do you score citation accuracy without human review on every release?
- Compute citation precision and recall mechanically on a labelled set where each claim has a known supporting span.
- Run the quote-existence check on all traffic because it costs microseconds, and sample the entailment check by stakes rather than by average.
- Keep a small human-labelled slice to calibrate the judge, and re-baseline when the judge model or prompt changes.
How do you stop near-duplicate policy versions from being confused when both are retrieved?
- Store version, effective date and supersedes links on every chunk, then filter to the version in force instead of trusting wording similarity.
- Delete superseded chunks at ingest rather than leaving both retrievable, and reconcile the index against source versions.
- When an archived copy must stay for audit, mark it non-retrievable for answering and cite the current version.
Red flags
- citation implies grounding
- existence-only check
Concepts this tests
Tags
More in RAG & retrieval pipelines
- Explain parent-child (small-to-big) retrieval. What problem does it solve and what does it cost you?
- Explain HNSW. What do M, ef_construction and ef_search each control, and which can you change in production?
- How would you tune HNSW for 10 million vectors when the product needs p95 retrieval under 100 ms?