Why it matters
Real query traffic is not homogeneous. One request is "what is the refund window", another is "ERR_2041 on checkout", a third is "SOC 2 Type II report 2025". A dense retriever handles the first well and can miss the second entirely: an error code is a rare token, and embedding models compress rare tokens toward the topics around them, so "ERR_2041" lands near other error discussions rather than near that specific code.
A lexical retriever such as BM25 does the opposite. It is precise on rare tokens and identifiers because term frequency and inverse document frequency reward exactly that, and it fails on paraphrase because it has no notion of synonymy. The two failure profiles are close to complementary, which is why most mature retrieval stacks run both.
Mental model
Two librarians answer the same request. One searches by meaning and brings back passages about the right topic. The other searches by the exact words on the page and brings back passages containing those words. You merge their shortlists and keep whichever books appear on both.
The metaphor breaks at the merge. Two librarians disagreeing is informative, but the merge rule throws information away: a book ranked first by both is obviously strong, while a book ranked first by one and fortieth by the other needs a policy. Rank fusion deliberately ignores how confident each librarian was, which is robust but blunt.
How it works
Synthetic index record · one ingest pass · two indexes
Seven chunk records · four in both indexes
bounding boxes
Step 1Ingest once, write twice
Chunks are parsed once and written to a lexical index and a vector index in the same pass. Both ids are derived from the chunk id, so a hit on either side resolves to one passage.
HYBRID INDEX · ONE SOURCE OF TRUTH · RUN 09:14Z
One chunk, two index ids
| Chunk | Lex | Vec | Hash | State |
|---|---|---|---|---|
| c-8841#3 | 3 | 3 | 9f2c… | both |
| c-8841#3b | 3b | 3b | 9f2c… | ≡dup |
| c-8841#4 | 4 | 4 | 71ab… | both |
| c-8841#5 | 5 | — | 38de… | !lex only |
| c-9012#1 | 1 | 1 | 5b07… | both |
| c-9012#2 | 2 | 2 | 22e1… | both |
| c-9012#3 | — | 3 | 4c9d… | !vec only |
2 indexes · 1 ingest pass · 7 chunks · 4 in both · 2 one-sided · 1 duplicate
Step record
- Indexes
- 2
- Ingest passes
- 1
- Ids per chunk
- 2
The text version summarises this figure; select a region for its full text.
Figure. One chunk written to a lexical index and a vector index in the same ingest pass. Of seven chunk records, four resolve in both indexes, one is lexical only and one vector only, and one is a duplicate text hash under a second id. The query returns nine hits over six unique chunks; the lexical-only chunk is dropped because there is no vector to display.
What to notice. Synthetic index record. Chunk ids, hashes and hit counts are illustrative. The figure shows the record a hybrid index has to keep so a hit can be cited, not a measured latency or a retrieval score; ranking is shown in the fusion lab below.
Text version
An index record for a hybrid retrieval run. Fields give the chunk id c-8841#3, its text hash 9f2c…4a1, and its lexical and vector ids. The chunk text records that ERR-2041 appears when a cancellation is attempted after the renewal date. A table lists seven chunks against both indexes: four appear in both, c-8841#3b repeats the same hash as a duplicate, c-8841#5 is lexical only and c-9012#3 is vector only. A query result says nine hits over six unique chunks, three in both lists, one dropped. Steps mark the two ids, the one-sided chunks, the duplicate, the query record, the drop, and the per-slice evaluation.
- Index twice, from one source of truth. Chunks go into a lexical index and a vector index in the same ingestion pass, with the same chunk IDs. Divergent pipelines are the most common cause of hybrid results that cite text you cannot display.
- Analyse text for lexical search. Tokenisation, stemming and stop-word choices matter for identifiers; aggressive stemming can collapse
ERR-2041andERR-2042. Test the analyser against real identifier queries. - Retrieve in parallel. Take the top 50–200 from each retriever. The candidate depth must be deep enough that fusion has material to work with; fusing two top-10 lists leaves little room.
- Fuse. Reciprocal rank fusion scores each document by across retrievers, which needs no score normalisation. Weighted fusion normalises scores and adds them, which preserves confidence but requires per-model calibration.
- Deduplicate. The same passage often arrives from both lists with different IDs after re-chunking; collapse by document and text hash.
- Rerank. A cross-encoder on the fused shortlist recovers precision that fusion cannot.
- Evaluate per slice. Report recall separately for identifier queries, natural-language questions and multi-constraint queries. A single aggregate hides the slice you broke.
BM25 for a document and query terms :
with the term frequency, the document length, and , controlling term-frequency saturation and length normalisation.
A minimal implementation
def rrf(lists: list[list[str]], k: int = 60, weights: list[float] | None = None) -> list[str]:
"""Fuse ranked ID lists. Ranks are 1-based; k damps the top of each list."""
weights = weights or [1.0] * len(lists)
scores: dict[str, float] = {}
for weight, ids in zip(weights, lists):
for rank, doc_id in enumerate(ids, start=1):
scores[doc_id] = scores.get(doc_id, 0.0) + weight / (k + rank)
return sorted(scores, key=scores.get, reverse=True)
def hybrid_search(query: str, k: int = 100) -> list[str]:
dense = vector_index.search(embed(query), k=k) # semantic
lexical = bm25_index.search(analyse(query), k=k) # exact tokens
return rrf([dense, lexical], k=60)[:50]Run the two searches concurrently. Sequentially, hybrid latency is the sum rather than the maximum, which is the usual reason a hybrid rollout misses its budget.
Query
ERR_2041 when cancelling my annual plan — do I still get a refund?
- 1. Retrieve both lists
- 2. Fuse by rank
- 3. Rerank the fused list
- 4. Assemble the prompt
BM25 · lexical
ranks exact tokens
- 1ERR_2041 checkout retry loopBM25 only
- 2Refund window — annual plans
- 3Billing FAQ: changing plans
- 4SOC 2 Type II report (2025)
- 5Cancelling a subscription
- 6Proration on plan upgrades
- 7Invoices and tax documents
- 8How a refund is processed
Dense · embeddings
ranks meaning
- 1Refund window — annual plans
- 2Money-back guarantee, 14 daysdense only
- 3How a refund is processed
- 4Cancelling a subscription
- 5Billing FAQ: changing plans
- 6Proration on plan upgrades
- 7Invoices and tax documents
- 8SOC 2 Type II report (2025)
The lists overlap on seven documents but agree on almost no order. Step on to fuse them by rank.
Fused scores are computed from the two ranks with 1/(60 + rank); rerank scores are illustrative. Fusion is by rank, not by score — the two retrievers fail differently, so a document only one of them returns is evidence, not noise.
Figure. Two retrievers rank eight documents each, overlapping on seven (nine distinct), fused by reciprocal rank fusion and then reranked. Fusion is by rank, not score: each list contributes 1/(60 + rank). With lists this short every shared document sits inside k = 60, so agreement dominates; longer candidate lists and smaller k behave differently. Rerank scores are invented for the demonstration.
What to notice. The error-code match appears only in the BM25 list and the paraphrase only in the dense list. RRF pushes both down to ranks 8 and 9 because it rewards agreement, not confidence; the cross-encoder then restores them. The two retrievers fail differently, which is the reason to run both — and fusion alone is not enough to keep what each one found.
Text version
A query asks about error ERR_2041 when cancelling an annual plan and whether a refund still applies. Two ranked lists of eight documents appear side by side. BM25 ranks the error-code page first, then the refund window, billing FAQ, SOC 2 report, cancellation page, proration page, invoices page and refund process page; the money-back guarantee passage is absent. The dense list ranks the refund window first, the money-back guarantee second, the refund process third, cancellation fourth, billing FAQ fifth, proration sixth, invoices seventh and the SOC 2 report eighth; the error-code page is absent. Seven documents appear in both lists, in almost no shared order. Fusing with reciprocal rank fusion, k = 60, produces nine rows: refund window, billing FAQ, cancelling a subscription, how a refund is processed, SOC 2, proration, invoices, then the error-code page at rank 8 and the money-back guarantee at rank 9, each marked as unique to one retriever. A cross-encoder rescores all nine and the order becomes: error code 0.94, money-back guarantee 0.91, refund window 0.88, billing FAQ 0.54, cancellation 0.49, refund process 0.41, proration 0.22, invoices 0.19, SOC 2 0.08. The two single-retriever documents move up seven places; the rest move down one, two or four places. The final step puts the top four passages into the context window, so the prompt holds both the exact error string and the refund policy sentence.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Dense only | Conversational, paraphrase-heavy traffic | Misses codes, names and rare tokens | Queries contain identifiers or exact phrases |
| Lexical only | Regulated text, identifiers, exact matches | No synonym handling | Users phrase questions naturally |
| Hybrid with RRF | Mixed traffic, no tuning data | Two indexes, two searches, a fusion step | One retriever is much stronger and should dominate |
| Hybrid with weighted scores | Mature systems with labelled queries | Score calibration per model and per domain | Scores drift after a model upgrade |
| Hybrid plus reranker | Hard queries where top-5 precision matters | Extra model call on the critical path | Latency budget is already tight |
Failure modes
- Fusion weight tuned on the average. Aggregate recall improves while identifier queries regress, because dense results dominate the top of the fused list. Debug: split the eval set by query type and compare recall per slice before and after the fusion change.
- Analyser destroys identifiers. Stemming or lowercasing collapses distinct codes. Debug: run five known error codes through the analyser and inspect the produced tokens; if two codes produce the same token stream, the index cannot separate them.
- Index divergence. Lexical results reference chunks that no longer exist in the vector store. Debug: join result IDs against the chunk table and count misses; alert when the miss rate is non-zero.
- Latency creep. Two sequential searches plus fusion push p95 over budget. Debug: trace each retrieval span separately; if the searches are serialised, fix that before reducing candidate depth.
In production
Hybrid search roughly doubles the retrieval work and adds a merge step, so the latency budget has to be explicit: parallel searches, one overall timeout, and a defined degraded mode (usually dense-only) if the lexical index is unavailable.
Monitoring should separate the two retrievers: hit rate and latency each, the overlap rate between their lists, and per-slice recall after fusion. A sudden rise in overlap often means the analyser changed and lexical search is now duplicating dense behaviour rather than complementing it.
Common misconceptions
People think: Hybrid always beats dense retrieval.
Actually: On clean, paraphrase-heavy corpora, lexical results add noise and latency. Hybrid wins where the query mix includes exact tokens or rare terms.
People think: RRF needs no tuning, so there is nothing to get wrong.
Actually: The damping constant and candidate depth still change results, and RRF discards score confidence entirely. It is a safe default, not a neutral one.
Interview angle
Interviewers often open with "why would you use BM25 in 2026?" The strong answer is not nostalgia: it is that inverse document frequency is a good prior for rare tokens, and dense models systematically blur them. Then describe fusion by rank, the eval split by query type, and the latency cost of running two indexes. See RAG interview questions and retrieval fusion.
Sources
- Cormack, Clarke & Buettcher, Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods — the RRF formulation used above.
- Robertson & Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond — derivation and parameters of BM25.
- Thakur et al., BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models — evidence that lexical baselines remain competitive out of domain.
- Elastic, Reciprocal rank fusion — how a production search engine exposes RRF over lexical and dense retrievers.