Skip to content
AI Engineer Guide.
seniorPracticecurated

What does the retriever do when a query has no relevant documents at all? Design the abstain path.

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

How do you calibrate an abstain threshold when you have no labelled unanswerable questions?
  • Mine candidates from production: queries whose best candidate scored low, and questions users rephrased after receiving an answer.
  • Generate unanswerable variants about entities or periods absent from the corpus, then hand-check a sample to confirm genuine absence.
  • Start permissive enough that false abstentions on answerable questions are measured, then tighten until misses fall, recording the miss versus over-block trade-off per category.
Should the system answer partially when one of three sub-claims is unsupported?
  • Answer the supported parts and state which part could not be supported, because a silent omission reads to the user as a complete answer.
  • Attribute evidence per sub-question so synthesis detects the gap instead of merging what it happens to have.
  • Abstain entirely when the missing part changes the decision, since a partial answer to a comparison misleads more than a refusal.
How do you tell a genuine no-answer from a retrieval miss, and does the user see the difference?
  • A genuine no-answer has no supporting passage in the corpus; a miss has one that search failed to surface, so run exact search before concluding.
  • Treat a miss as a retrieval bug with a fix and a genuine no-answer as an abstention with a product response.
  • Internally the two carry different reason codes, while the user sees one honest message because the distinction is not actionable for them.

Red flags

  • always answers
  • threshold by vibes

Concepts this tests

Tags

More in RAG & retrieval pipelines