Why it matters
A follow-up question like 'and for the enterprise tier?' carries its meaning in the conversation, not in its words. Sent to a retriever as-is, it matches almost nothing. Query rewriting reconstructs the standalone information need from dialogue state before retrieval, which raises recall without touching the index — but it introduces a model in the critical path whose errors are invisible unless the original and rewritten forms are both logged.
Mental model
Rewriting is a reference librarian restating a mumbled request as a precise one before searching. The restatement helps, and it also encodes the librarian's assumptions, which may not be the asker's. It breaks when the correction is confident and wrong, because the search then answers a question nobody asked.
How it works
- 1Detect whether the query is standalone or depends on prior turns, using pronoun and ellipsis signals rather than always rewriting.
- 2For dependent queries, resolve references against the conversation state and produce one explicit standalone question.
- 3Normalise spelling, abbreviations and units that the corpus spells differently, using a curated map before a model.
- 4Preserve identifiers verbatim; never let a rewrite paraphrase a part number or error code.
- 5Log the original, the rewritten form and the retrieved ids together so a bad rewrite is diagnosable.
- 6Evaluate recall on the rewritten query and on the original separately to confirm the rewrite is the reason for the gain.
Step 1 of 6
Detect whether the query is standalone or depends on prior turns, using pronoun and ellipsis signals rather than always rewriting.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Detect whether the query is standalone or depends on prior turns, using pronoun and ellip… 2. For dependent queries, resolve references against the conversation state and produce one … 3. Normalise spelling, abbreviations and units that the corpus spells differently, using a c… 4. Preserve identifiers verbatim; never let a rewrite paraphrase a part number or error code. 5. Log the original, the rewritten form and the retrieved ids together so a bad rewrite is d… 6. Evaluate recall on the rewritten query and on the original separately to confirm the rewr…
- Detect whether the query is standalone or depends on prior turns, using pronoun and ellipsis signals rather than always rewriting.
- For dependent queries, resolve references against the conversation state and produce one explicit standalone question.
- Normalise spelling, abbreviations and units that the corpus spells differently, using a curated map before a model.
- Preserve identifiers verbatim; never let a rewrite paraphrase a part number or error code.
- Log the original, the rewritten form and the retrieved ids together so a bad rewrite is diagnosable.
- Evaluate recall on the rewritten query and on the original separately to confirm the rewrite is the reason for the gain.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Always rewrite with a model | Handles every conversational case | Latency and a new failure mode on simple queries | Queries are already explicit and latency-critical |
| Rewrite only dependent turns | Cheap and low risk | Needs dependency detection that can itself be wrong | Conversations switch topics without clear markers |
| Rule-based normalisation | Deterministic and fast | Limited to known abbreviations and typos | Users use unanticipated shorthand |
| Rewrite plus original in one search | Hedges against drift | Doubles retrieval cost | The index is large and latency is tight |
Good for
Handles every conversational case
Cost
Latency and a new failure mode on simple queries
Fails when
Queries are already explicit and latency-critical
- Queries are already explicit and latency-critical
- Conversations switch topics without clear markers
- Users use unanticipated shorthand
- The index is large and latency is tight
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Always rewrite with a model is good for Handles every conversational case, costs Latency and a new failure mode on simple queries, and fails when Queries are already explicit and latency-critical. Rewrite only dependent turns is good for Cheap and low risk, costs Needs dependency detection that can itself be wrong, and fails when Conversations switch topics without clear markers. Rule-based normalisation is good for Deterministic and fast, costs Limited to known abbreviations and typos, and fails when Users use unanticipated shorthand. Rewrite plus original in one search is good for Hedges against drift, costs Doubles retrieval cost, and fails when The index is large and latency is tight
Failure modes
- Over-specific rewrite. The model adds a qualifier the user never said, and retrieval narrows to the wrong subset. Debug: diff original and rewritten queries for added entities or constraints.
- Identifier paraphrased. A model rewrites a product code into a description and the exact match is lost. Debug: assert that tokens matching the identifier pattern appear unchanged in the rewrite.
- History contamination. A stale topic from earlier turns leaks into an unrelated new question. Debug: read the rewrite beside the last three turns and check which turn supplied each entity.
In production
Log rewrite latency and the share of rewrites that change the top result. Cap rewrite length, and fall back to the original query when the rewrite returns nothing.
Common misconceptions
People think: Rewriting always improves recall.
Actually: It improves recall for dependent or malformed queries and can reduce it by adding constraints.
People think: The rewritten query should replace the original everywhere.
Actually: The original is the only faithful record of intent and the safest fallback when the rewrite fails.
Interview angle
Distinguish standalone from dependent queries and name the risk of drift on identifiers. Insist on logging both forms and evaluating the gain. Asking a model to rewrite the query, with no fallback and no way to tell whether the rewrite helped, is the thin answer.
Sources
- Ma et al., Query Rewriting for Retrieval-Augmented Large Language Models — rewrite-retrieve-read and the effect of rewriting on retrieval.
- Gao et al., Precise Zero-Shot Dense Retrieval without Relevance Labels — generating a retrieval target when the query itself is a poor match.
- Karpukhin et al., Dense Passage Retrieval — the query encoder whose behaviour rewriting is trying to satisfy.