Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 29 of 57

Query understanding

0 of 57 complete

Why it matters

A follow-up question like 'and for the enterprise tier?' carries its meaning in the conversation, not in its words. Sent to a retriever as-is, it matches almost nothing. Query rewriting reconstructs the standalone information need from dialogue state before retrieval, which raises recall without touching the index — but it introduces a model in the critical path whose errors are invisible unless the original and rewritten forms are both logged.

Mental model

Rewriting is a reference librarian restating a mumbled request as a precise one before searching. The restatement helps, and it also encodes the librarian's assumptions, which may not be the asker's. It breaks when the correction is confident and wrong, because the search then answers a question nobody asked.

How it works

Mechanism, step by stepInteractive
  1. 1Detect whether the query is standalone or depends on prior turns, using pronoun and ellipsis signals rather than always rewriting.
  2. 2For dependent queries, resolve references against the conversation state and produce one explicit standalone question.
  3. 3Normalise spelling, abbreviations and units that the corpus spells differently, using a curated map before a model.
  4. 4Preserve identifiers verbatim; never let a rewrite paraphrase a part number or error code.
  5. 5Log the original, the rewritten form and the retrieved ids together so a bad rewrite is diagnosable.
  6. 6Evaluate recall on the rewritten query and on the original separately to confirm the rewrite is the reason for the gain.

Step 1 of 6

Detect whether the query is standalone or depends on prior turns, using pronoun and ellipsis signals rather than always rewriting.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Detect whether the query is standalone or depends on prior turns, using pronoun and ellip… 2. For dependent queries, resolve references against the conversation state and produce one … 3. Normalise spelling, abbreviations and units that the corpus spells differently, using a c… 4. Preserve identifiers verbatim; never let a rewrite paraphrase a part number or error code. 5. Log the original, the rewritten form and the retrieved ids together so a bad rewrite is d… 6. Evaluate recall on the rewritten query and on the original separately to confirm the rewr…

  1. Detect whether the query is standalone or depends on prior turns, using pronoun and ellipsis signals rather than always rewriting.
  2. For dependent queries, resolve references against the conversation state and produce one explicit standalone question.
  3. Normalise spelling, abbreviations and units that the corpus spells differently, using a curated map before a model.
  4. Preserve identifiers verbatim; never let a rewrite paraphrase a part number or error code.
  5. Log the original, the rewritten form and the retrieved ids together so a bad rewrite is diagnosable.
  6. Evaluate recall on the rewritten query and on the original separately to confirm the rewrite is the reason for the gain.

Trade-offs

ChoiceGood forCostFails when
Always rewrite with a modelHandles every conversational caseLatency and a new failure mode on simple queriesQueries are already explicit and latency-critical
Rewrite only dependent turnsCheap and low riskNeeds dependency detection that can itself be wrongConversations switch topics without clear markers
Rule-based normalisationDeterministic and fastLimited to known abbreviations and typosUsers use unanticipated shorthand
Rewrite plus original in one searchHedges against driftDoubles retrieval costThe index is large and latency is tight
Trade-off mapInteractive

Good for

Handles every conversational case

Cost

Latency and a new failure mode on simple queries

Fails when

Queries are already explicit and latency-critical

  1. Queries are already explicit and latency-critical
  2. Conversations switch topics without clear markers
  3. Users use unanticipated shorthand
  4. The index is large and latency is tight

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Always rewrite with a model is good for Handles every conversational case, costs Latency and a new failure mode on simple queries, and fails when Queries are already explicit and latency-critical. Rewrite only dependent turns is good for Cheap and low risk, costs Needs dependency detection that can itself be wrong, and fails when Conversations switch topics without clear markers. Rule-based normalisation is good for Deterministic and fast, costs Limited to known abbreviations and typos, and fails when Users use unanticipated shorthand. Rewrite plus original in one search is good for Hedges against drift, costs Doubles retrieval cost, and fails when The index is large and latency is tight

Failure modes

  • Over-specific rewrite. The model adds a qualifier the user never said, and retrieval narrows to the wrong subset. Debug: diff original and rewritten queries for added entities or constraints.
  • Identifier paraphrased. A model rewrites a product code into a description and the exact match is lost. Debug: assert that tokens matching the identifier pattern appear unchanged in the rewrite.
  • History contamination. A stale topic from earlier turns leaks into an unrelated new question. Debug: read the rewrite beside the last three turns and check which turn supplied each entity.

In production

Log rewrite latency and the share of rewrites that change the top result. Cap rewrite length, and fall back to the original query when the rewrite returns nothing.

Common misconceptions

People think: Rewriting always improves recall.

Actually: It improves recall for dependent or malformed queries and can reduce it by adding constraints.

People think: The rewritten query should replace the original everywhere.

Actually: The original is the only faithful record of intent and the safest fallback when the rewrite fails.

Interview angle

Distinguish standalone from dependent queries and name the risk of drift on identifiers. Insist on logging both forms and evaluating the gain. Asking a model to rewrite the query, with no fallback and no way to tell whether the rewrite helped, is the thin answer.

Sources