Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 31 of 57

Multi-query

0 of 57 complete

Why it matters

One query is one sample from the space of ways to express a need. If the corpus phrases the topic differently, that sample misses. Multi-query retrieval draws several samples and unions the results, which raises the chance that the right passage appears in at least one list. The engineering question is not whether diversity helps but whether the extra retrieval calls earn their latency on your query distribution.

Mental model

Multi-query is asking the same question to several colleagues who each interpret it slightly differently, then pooling their answers. The metaphor breaks because the colleagues share your blind spots if they were generated from your phrasing, so the pool can be wide and still miss.

How it works

Mechanism, step by stepInteractive
  1. 1Generate a small fixed number of variants that differ in vocabulary and specificity, not in meaning.
  2. 2Reject variants that are near-duplicates of the original by token overlap, to avoid paying for the same search twice.
  3. 3Retrieve for the original and each variant, keeping per-variant ranked lists.
  4. 4Fuse the lists by rank with reciprocal rank fusion, deduplicating by document id.
  5. 5Cap variants by latency budget and reduce them for queries where the first variant already scores well.
  6. 6Measure recall of the union against the single-query baseline on a labelled set before shipping.

Step 1 of 6

Generate a small fixed number of variants that differ in vocabulary and specificity, not in meaning.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Generate a small fixed number of variants that differ in vocabulary and specificity, not … 2. Reject variants that are near-duplicates of the original by token overlap, to avoid payin… 3. Retrieve for the original and each variant, keeping per-variant ranked lists. 4. Fuse the lists by rank with reciprocal rank fusion, deduplicating by document id. 5. Cap variants by latency budget and reduce them for queries where the first variant alread… 6. Measure recall of the union against the single-query baseline on a labelled set before sh…

  1. Generate a small fixed number of variants that differ in vocabulary and specificity, not in meaning.
  2. Reject variants that are near-duplicates of the original by token overlap, to avoid paying for the same search twice.
  3. Retrieve for the original and each variant, keeping per-variant ranked lists.
  4. Fuse the lists by rank with reciprocal rank fusion, deduplicating by document id.
  5. Cap variants by latency budget and reduce them for queries where the first variant already scores well.
  6. Measure recall of the union against the single-query baseline on a labelled set before shipping.

Trade-offs

ChoiceGood forCostFails when
Three variantsBroad vocabulary coverageThree times retrieval latency and costTraffic is high and the budget is per-request
Many variantsMaximum recall on rare phrasingDiminishing returns and noisy fusionLatency or rate limits bind
Variants from an LLMNatural paraphrase and expansionModel latency and occasional driftQueries are already precise identifiers
Single query with fusionCheapestMisses vocabulary the query never usedUsers and corpus use different registers
Trade-off mapInteractive

Good for

Broad vocabulary coverage

Cost

Three times retrieval latency and cost

Fails when

Traffic is high and the budget is per-request

  1. Traffic is high and the budget is per-request
  2. Latency or rate limits bind
  3. Queries are already precise identifiers
  4. Users and corpus use different registers

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Three variants is good for Broad vocabulary coverage, costs Three times retrieval latency and cost, and fails when Traffic is high and the budget is per-request. Many variants is good for Maximum recall on rare phrasing, costs Diminishing returns and noisy fusion, and fails when Latency or rate limits bind. Variants from an LLM is good for Natural paraphrase and expansion, costs Model latency and occasional drift, and fails when Queries are already precise identifiers. Single query with fusion is good for Cheapest, costs Misses vocabulary the query never used, and fails when Users and corpus use different registers

Failure modes

  • Duplicate variants. Three paraphrases produce almost the same list, tripling cost for no recall gain. Debug: compute pairwise Jaccard overlap of the variants and of their result lists.
  • Semantic drift in a variant. One paraphrase asks a different question and injects irrelevant passages that survive fusion. Debug: label each retrieved passage with the variant that found it and inspect the odd one out.
  • Fusion without deduplication. The same passage appears once per variant and fills the context. Debug: count distinct document ids in the fused top-k.

In production

Track recall gain per variant and drop variants that never contribute a unique relevant result. Cap the variant count and cache results keyed by the original query hash.

Common misconceptions

People think: More paraphrases always raise recall.

Actually: Gain comes from vocabulary diversity; redundant variants only add latency.

People think: Multi-query replaces query rewriting.

Actually: Rewriting makes one better query; multi-query hedges across several, and they compose.

Interview angle

Frame multi-query as recall insurance priced in latency, require diversity between variants, and describe fusion with deduplication. Name the measurement that justifies keeping a variant. Generating five queries and concatenating the results is the answer that triples cost for no recall.

Sources