Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 15 of 57

Hybrid search

0 of 57 complete

Why it matters

Real query traffic is not homogeneous. One request is "what is the refund window", another is "ERR_2041 on checkout", a third is "SOC 2 Type II report 2025". A dense retriever handles the first well and can miss the second entirely: an error code is a rare token, and embedding models compress rare tokens toward the topics around them, so "ERR_2041" lands near other error discussions rather than near that specific code.

A lexical retriever such as BM25 does the opposite. It is precise on rare tokens and identifiers because term frequency and inverse document frequency reward exactly that, and it fails on paraphrase because it has no notion of synonymy. The two failure profiles are close to complementary, which is why most mature retrieval stacks run both.

Mental model

Two librarians answer the same request. One searches by meaning and brings back passages about the right topic. The other searches by the exact words on the page and brings back passages containing those words. You merge their shortlists and keep whichever books appear on both.

The metaphor breaks at the merge. Two librarians disagreeing is informative, but the merge rule throws information away: a book ranked first by both is obviously strong, while a book ranked first by one and fortieth by the other needs a policy. Rank fusion deliberately ignores how confident each librarian was, which is robust but blunt.

How it works

One chunk, two index idsInteractive

Synthetic index record · one ingest pass · two indexes

Seven chunk records · four in both indexes

bounding boxes

Step 1/6 · Ingest once, write twice

Step 1Ingest once, write twice

Chunks are parsed once and written to a lexical index and a vector index in the same pass. Both ids are derived from the chunk id, so a hit on either side resolves to one passage.

HYBRID INDEX · ONE SOURCE OF TRUTH · RUN 09:14Z

, running header, region 1. Select to inspect.

One chunk, two index ids

, title, region 2. Select to inspect.
Chunkc-8841#3
, form field Chunk, region 3. Select to inspect.
Text hash9f2c…4a1
, form field Text hash, region 4. Select to inspect.
Lexical idlex:8841#3
, form field Lexical id, region 5. Select to inspect.
Vector idvec:8841#3
, form field Vector id, region 6. Select to inspect.
Chunk text

3 · Cancellation — ERR_2041 appears when a cancellation is attempted after the renewal date; annual plans keep a 14-day window.

, paragraph Chunk text, region 7. Select to inspect.
INDEX RECORD
ChunkLexVecHashState
c-8841#3339f2c…both
c-8841#3b3b3b9f2c…≡dup
c-8841#44471ab…both
c-8841#55—38de…!lex only
c-9012#1115b07…both
c-9012#22222e1…both
c-9012#3—34c9d…!vec only
, table INDEX RECORD, region 8. Select to inspect.
Query result9 hits · 6 unique chunks · 3 in both lists · 1 dropped
, form field Query result, region 9. Select to inspect.
Fail closed

A lexical hit whose chunk has no vector id cannot be shown as evidence. Drop it and count it; do not cite a passage the reader cannot open.

, paragraph Fail closed, region 10. Select to inspect.

2 indexes · 1 ingest pass · 7 chunks · 4 in both · 2 one-sided · 1 duplicate

, running footer, region 11. Select to inspect.
11 regions

Step record

Indexes
2
Ingest passes
1
Ids per chunk
2

The text version summarises this figure; select a region for its full text.

Marksduplicate contentwarning

Figure. One chunk written to a lexical index and a vector index in the same ingest pass. Of seven chunk records, four resolve in both indexes, one is lexical only and one vector only, and one is a duplicate text hash under a second id. The query returns nine hits over six unique chunks; the lexical-only chunk is dropped because there is no vector to display.

What to notice. Synthetic index record. Chunk ids, hashes and hit counts are illustrative. The figure shows the record a hybrid index has to keep so a hit can be cited, not a measured latency or a retrieval score; ranking is shown in the fusion lab below.

Text version

An index record for a hybrid retrieval run. Fields give the chunk id c-8841#3, its text hash 9f2c…4a1, and its lexical and vector ids. The chunk text records that ERR-2041 appears when a cancellation is attempted after the renewal date. A table lists seven chunks against both indexes: four appear in both, c-8841#3b repeats the same hash as a duplicate, c-8841#5 is lexical only and c-9012#3 is vector only. A query result says nine hits over six unique chunks, three in both lists, one dropped. Steps mark the two ids, the one-sided chunks, the duplicate, the query record, the drop, and the per-slice evaluation.

  1. Index twice, from one source of truth. Chunks go into a lexical index and a vector index in the same ingestion pass, with the same chunk IDs. Divergent pipelines are the most common cause of hybrid results that cite text you cannot display.
  2. Analyse text for lexical search. Tokenisation, stemming and stop-word choices matter for identifiers; aggressive stemming can collapse ERR-2041 and ERR-2042. Test the analyser against real identifier queries.
  3. Retrieve in parallel. Take the top 50–200 from each retriever. The candidate depth must be deep enough that fusion has material to work with; fusing two top-10 lists leaves little room.
  4. Fuse. Reciprocal rank fusion scores each document by ∑i1/(k+ranki)\sum_i 1/(k + \text{rank}_i) across retrievers, which needs no score normalisation. Weighted fusion normalises scores and adds them, which preserves confidence but requires per-model calibration.
  5. Deduplicate. The same passage often arrives from both lists with different IDs after re-chunking; collapse by document and text hash.
  6. Rerank. A cross-encoder on the fused shortlist recovers precision that fusion cannot.
  7. Evaluate per slice. Report recall separately for identifier queries, natural-language questions and multi-constraint queries. A single aggregate hides the slice you broke.

BM25 for a document DD and query terms tt:

score(D,Q)=∑t∈QIDF(t)⋅f(t,D) (k1+1)f(t,D)+k1(1−b+b ∣D∣avgdl)\text{score}(D, Q) = \sum_{t \in Q} \text{IDF}(t) \cdot \frac{f(t,D)\,(k_1 + 1)}{f(t,D) + k_1\left(1 - b + b\,\frac{|D|}{\text{avgdl}}\right)}

with f(t,D)f(t,D) the term frequency, ∣D∣|D| the document length, and k1k_1, bb controlling term-frequency saturation and length normalisation.

A minimal implementation

def rrf(lists: list[list[str]], k: int = 60, weights: list[float] | None = None) -> list[str]:
    """Fuse ranked ID lists. Ranks are 1-based; k damps the top of each list."""
    weights = weights or [1.0] * len(lists)
    scores: dict[str, float] = {}
    for weight, ids in zip(weights, lists):
        for rank, doc_id in enumerate(ids, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + weight / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)
 
def hybrid_search(query: str, k: int = 100) -> list[str]:
    dense = vector_index.search(embed(query), k=k)      # semantic
    lexical = bm25_index.search(analyse(query), k=k)    # exact tokens
    return rrf([dense, lexical], k=60)[:50]

Run the two searches concurrently. Sequentially, hybrid latency is the sum rather than the maximum, which is the usual reason a hybrid rollout misses its budget.

Fusion lab · BM25 + dense → RRF → rerankInteractive

Query

ERR_2041 when cancelling my annual plan — do I still get a refund?

  1. 1. Retrieve both lists
  2. 2. Fuse by rank
  3. 3. Rerank the fused list
  4. 4. Assemble the prompt

BM25 · lexical

ranks exact tokens

  1. 1ERR_2041 checkout retry loopBM25 only
  2. 2Refund window — annual plans
  3. 3Billing FAQ: changing plans
  4. 4SOC 2 Type II report (2025)
  5. 5Cancelling a subscription
  6. 6Proration on plan upgrades
  7. 7Invoices and tax documents
  8. 8How a refund is processed

Dense · embeddings

ranks meaning

  1. 1Refund window — annual plans
  2. 2Money-back guarantee, 14 daysdense only
  3. 3How a refund is processed
  4. 4Cancelling a subscription
  5. 5Billing FAQ: changing plans
  6. 6Proration on plan upgrades
  7. 7Invoices and tax documents
  8. 8SOC 2 Type II report (2025)

The lists overlap on seven documents but agree on almost no order. Step on to fuse them by rank.

Step 1/4 · Retrieve both lists

Fused scores are computed from the two ranks with 1/(60 + rank); rerank scores are illustrative. Fusion is by rank, not by score — the two retrievers fail differently, so a document only one of them returns is evidence, not noise.

Figure. Two retrievers rank eight documents each, overlapping on seven (nine distinct), fused by reciprocal rank fusion and then reranked. Fusion is by rank, not score: each list contributes 1/(60 + rank). With lists this short every shared document sits inside k = 60, so agreement dominates; longer candidate lists and smaller k behave differently. Rerank scores are invented for the demonstration.

What to notice. The error-code match appears only in the BM25 list and the paraphrase only in the dense list. RRF pushes both down to ranks 8 and 9 because it rewards agreement, not confidence; the cross-encoder then restores them. The two retrievers fail differently, which is the reason to run both — and fusion alone is not enough to keep what each one found.

Text version

A query asks about error ERR_2041 when cancelling an annual plan and whether a refund still applies. Two ranked lists of eight documents appear side by side. BM25 ranks the error-code page first, then the refund window, billing FAQ, SOC 2 report, cancellation page, proration page, invoices page and refund process page; the money-back guarantee passage is absent. The dense list ranks the refund window first, the money-back guarantee second, the refund process third, cancellation fourth, billing FAQ fifth, proration sixth, invoices seventh and the SOC 2 report eighth; the error-code page is absent. Seven documents appear in both lists, in almost no shared order. Fusing with reciprocal rank fusion, k = 60, produces nine rows: refund window, billing FAQ, cancelling a subscription, how a refund is processed, SOC 2, proration, invoices, then the error-code page at rank 8 and the money-back guarantee at rank 9, each marked as unique to one retriever. A cross-encoder rescores all nine and the order becomes: error code 0.94, money-back guarantee 0.91, refund window 0.88, billing FAQ 0.54, cancellation 0.49, refund process 0.41, proration 0.22, invoices 0.19, SOC 2 0.08. The two single-retriever documents move up seven places; the rest move down one, two or four places. The final step puts the top four passages into the context window, so the prompt holds both the exact error string and the refund policy sentence.

Trade-offs

ChoiceGood forCostFails when
Dense onlyConversational, paraphrase-heavy trafficMisses codes, names and rare tokensQueries contain identifiers or exact phrases
Lexical onlyRegulated text, identifiers, exact matchesNo synonym handlingUsers phrase questions naturally
Hybrid with RRFMixed traffic, no tuning dataTwo indexes, two searches, a fusion stepOne retriever is much stronger and should dominate
Hybrid with weighted scoresMature systems with labelled queriesScore calibration per model and per domainScores drift after a model upgrade
Hybrid plus rerankerHard queries where top-5 precision mattersExtra model call on the critical pathLatency budget is already tight

Failure modes

  • Fusion weight tuned on the average. Aggregate recall improves while identifier queries regress, because dense results dominate the top of the fused list. Debug: split the eval set by query type and compare recall per slice before and after the fusion change.
  • Analyser destroys identifiers. Stemming or lowercasing collapses distinct codes. Debug: run five known error codes through the analyser and inspect the produced tokens; if two codes produce the same token stream, the index cannot separate them.
  • Index divergence. Lexical results reference chunks that no longer exist in the vector store. Debug: join result IDs against the chunk table and count misses; alert when the miss rate is non-zero.
  • Latency creep. Two sequential searches plus fusion push p95 over budget. Debug: trace each retrieval span separately; if the searches are serialised, fix that before reducing candidate depth.

In production

Hybrid search roughly doubles the retrieval work and adds a merge step, so the latency budget has to be explicit: parallel searches, one overall timeout, and a defined degraded mode (usually dense-only) if the lexical index is unavailable.

Monitoring should separate the two retrievers: hit rate and latency each, the overlap rate between their lists, and per-slice recall after fusion. A sudden rise in overlap often means the analyser changed and lexical search is now duplicating dense behaviour rather than complementing it.

Common misconceptions

People think: Hybrid always beats dense retrieval.

Actually: On clean, paraphrase-heavy corpora, lexical results add noise and latency. Hybrid wins where the query mix includes exact tokens or rare terms.

People think: RRF needs no tuning, so there is nothing to get wrong.

Actually: The damping constant and candidate depth still change results, and RRF discards score confidence entirely. It is a safe default, not a neutral one.

Interview angle

Interviewers often open with "why would you use BM25 in 2026?" The strong answer is not nostalgia: it is that inverse document frequency is a good prior for rare tokens, and dense models systematically blur them. Then describe fusion by rank, the eval split by query type, and the latency cost of running two indexes. See RAG interview questions and retrieval fusion.

Sources