Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 4 of 57

BM25

0 of 57 complete

Why it matters

Raw term counts reward documents that repeat a word and documents that are simply long. BM25 corrects both with two ideas: a term's contribution grows quickly and then flattens, and a document's score is discounted by how much longer it is than average. The result is a scoring function with no training data that remains hard to beat on exact-match retrieval, which is why it is the default first stage in most hybrid stacks.

Mental model

BM25 is a witness scoring system: a witness who mentions the detail twice is more credible than one who never does, but a tenth mention adds little, and a witness who rambles for an hour is discounted for length. The metaphor breaks because BM25 treats every mention as independent and cannot tell a confirmation from a repetition.

How it works

A document's score, term by termInteractive

Synthetic scored record · index 07 · query “refund window”

One document, two query terms, k1 1.2 · b 0.75

bounding boxes

Step 1/6 · Read the postings

Step 1Read the postings

For each query term the index returns how often it occurs in this document and in how many documents of the collection. Nothing is weighted yet: frequency and document frequency are the raw record.

LEXICAL INDEX · SCORED RECORD · QUERY: REFUND WINDOW

, running header, region 1. Select to inspect.

refund-policy · scored record

, title, region 2. Select to inspect.
Documentrefund-policy
, form field Document, region 3. Select to inspect.
Length96 tokens
, form field Length, region 4. Select to inspect.
Average120 tokens
, form field Average, region 5. Select to inspect.
Documents1,200
, form field Documents, region 6. Select to inspect.
QUERY TERMS
TermtfdfidfWeight
refund3483.215.27
window12101.741.90
total7.17
, table QUERY TERMS, region 7. Select to inspect.
Score5.27 + 1.90 = 7.17 · k1 1.2 · b 0.75
, form field Score, region 8. Select to inspect.
SAME TERMS
DocumentLengthrefundwindowScore
refund-policy965.271.907.17
long-report2404.151.245.39
, table SAME TERMS, region 9. Select to inspect.
SATURATION
refund at tf123510
weight3.504.685.275.866.41
, table SATURATION, region 10. Select to inspect.
Statistics

Document frequencies come from the current snapshot. After a bulk load the rare terms are less rare, so every stale idf overstates them.

, paragraph Statistics, region 11. Select to inspect.

df 48 · 210 · k1 1.2 · b 0.75 · 2 terms · score 7.17 · 1,200 documents

, running footer, region 12. Select to inspect.
12 regions

Step record

Documents
1,200
Average length
120 tokens
Terms
2

The text version summarises this figure; select a region for its full text.

Figure. One document scored for the query “refund window”: the postings give term frequency and document frequency, idf weights the rare term, and the two weights sum to 7.17. A second document carries the same term frequencies at 240 tokens, twice the corpus average, and scores 5.39 — the length correction on its own. Parameters k1 = 1.2 and b = 0.75 throughout.

What to notice. Synthetic corpus of 1,200 documents. The frequencies, idfs and weights are computed from the stated formula and rounded to two decimals; the corpus is not a measured collection. Scores are comparable only within one statistics snapshot, so the figure shows one snapshot.

Text version

A scored record for the document refund-policy, length 96 tokens, in a corpus of 1,200 documents averaging 120 tokens. Query terms refund and window have term frequencies 3 and 1, document frequencies 48 and 210, idf 3.21 and 1.74, and weights 5.27 and 1.90, summing to 7.17. A comparison table scores the same terms in a 240-token document at 4.15 and 1.24, totalling 5.39. A saturation table gives refund's weight at term frequencies 1, 2, 3, 5 and 10 as 3.50, 4.68, 5.27, 5.86 and 6.41. Steps mark the postings, the idf weights, saturation, length normalisation, the sum, and the statistics snapshot. Selecting a region shows its note.

  1. Build the term statistics first: document frequency per term, document length and average length across the collection.
  2. Compute the inverse document frequency for each query term from the current corpus, not from a stale snapshot.
  3. Sum the per-term contributions, applying saturation with k1 so repeated terms stop dominating.
  4. Apply length normalisation with b, choosing a value between no penalty and full penalty for your document length variance.
  5. Tune k1 and b only against a judged query set, and only after the analyzer and tokenization are stable.
  6. Compare against a dense retriever on identifier-heavy and paraphrase-heavy query slices separately.

Trade-offs

ChoiceGood forCostFails when
Default parametersReasonable quality with no tuningSuboptimal on unusual length distributionsDocuments vary wildly in length or are mostly short fields
Strong length normalisationFair ranking across long and short documentsDemotes long documents that genuinely cover the topicRelevant evidence lives in long reports
High k1Repeated terms keep contributingKeyword-stuffed documents climbDocuments repeat boilerplate
BM25 aloneExact terms, no training, explainable scoresFails on synonymy and paraphraseUsers query in vocabulary the corpus does not use
Trade-off mapInteractive

Good for

Reasonable quality with no tuning

Cost

Suboptimal on unusual length distributions

Fails when

Documents vary wildly in length or are mostly short fields

  1. Documents vary wildly in length or are mostly short fields
  2. Relevant evidence lives in long reports
  3. Documents repeat boilerplate
  4. Users query in vocabulary the corpus does not use

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Default parameters is good for Reasonable quality with no tuning, costs Suboptimal on unusual length distributions, and fails when Documents vary wildly in length or are mostly short fields. Strong length normalisation is good for Fair ranking across long and short documents, costs Demotes long documents that genuinely cover the topic, and fails when Relevant evidence lives in long reports. High k1 is good for Repeated terms keep contributing, costs Keyword-stuffed documents climb, and fails when Documents repeat boilerplate. BM25 alone is good for Exact terms, no training, explainable scores, costs Fails on synonymy and paraphrase, and fails when Users query in vocabulary the corpus does not use

Failure modes

  • Stale document frequencies. Scores drift after a large ingest because idf still reflects the old corpus. Debug: recompute term statistics and compare the idf of a known rare term before and after.
  • Length normalisation on the wrong unit. b is applied to a chunk while statistics were computed per page, so short chunks are penalised twice. Debug: print document length and average length for the top results and check the units match the index.
  • Score comparison across fields. A title field and a body field are scored with one scale, so titles dominate. Debug: log per-field contributions for a query where the correct answer is in the body.

In production

Log the score distribution and the per-term contributions for queries that fail. Recompute corpus statistics after bulk changes, and keep k1 and b in configuration so a rebuild uses the same values.

Common misconceptions

People think: BM25 is a legacy baseline.

Actually: It remains the strongest cheap first stage for exact-match retrieval and the lexical half of most hybrid systems.

People think: Higher term frequency always means a higher score.

Actually: Saturation and length normalisation mean a long, repetitive document can lose to a short, focused one.

Interview angle

Explain saturation, inverse document frequency and length normalisation as three distinct corrections, and say which parameter controls which. Then name the synonymy limit that motivates hybrid retrieval. Answers that reproduce the formula but cannot say what happens when b is zero, or why idf must be recomputed after a bulk load, are reciting rather than reasoning.

Sources