BM25 and Sparse Lexical Retrieval
No embeddings, no neural network — just term counts, weighted by how rare each term is and how long each document is. Still the backbone of most production search.
Parameters
query: "python list methods"
Ranked Documents
—Top Document, Term By Term
Corpus Stats
Overview
What this is
Before dense vector search, and still running alongside it in most real systems, is sparse lexical search. Score a document by which query terms it holds, weighted by how informative each term is. BM25 is the version of this idea that really works well. And it needs no training, no GPU, and no embedding model. BM25 scores a document by summing a term-weight for each query word that appears in it. The term weight combines two ideas: IDF (inverse document count) gives rare words more weight than common ones, and TF saturation ensures that a word appearing 10 times scores only slightly higher than one appearing 5 times. Preventing keyword-stuffed documents from ruling. The saturation is controlled by the parameter k1 (often 1.2-2.0). Higher values let term count matter more, lower values flatten the curve earlier. The parameter b (often 0.75) controls length normalisation: b=1 fully penalises long documents, b=0 ignores length. The "sparse" in sparse retrieval means the document form. Each document is a vector with one dimension per vocabulary term. And most entries are zero because any given document uses only a tiny fraction of the vocabulary. An inverted index stores only the non-zero entries. This makes retrieval fast even over millions of documents. Hybrid search combines BM25's lexical precision with dense vector search's semantic grasp. This page scores a set of documents against a query, shows the IDF and TF-saturation parts, and covers the k1/b parameter tradeoff.
No embeddings, no neural network — just term counts, weighted by how rare each term is and how long each document is.
BM25: A Practical Guide
Keyword search, done properly.
What this is
Before dense vector search, and still running alongside it in most real systems, is sparse lexical search. Score a document by which query terms it holds, weighted by how informative each term is. BM25 is the version of this idea that really works well. And it needs no training, no GPU, and no embedding model. BM25 scores a document by summing a term-weight for each query word that appears in it. The term weight combines two ideas: IDF (inverse document count) gives rare words more weight than common ones, and TF saturation ensures that a word appearing 10 times scores only slightly higher than one appearing 5 times. Preventing keyword-stuffed documents from ruling. The saturation is controlled by the parameter k1 (often 1.2-2.0). Higher values let term count matter more, lower values flatten the curve earlier. The parameter b (often 0.75) controls length normalisation: b=1 fully penalises long documents, b=0 ignores length. The "sparse" in sparse retrieval means the document form. Each document is a vector with one dimension per vocabulary term. And most entries are zero because any given document uses only a tiny fraction of the vocabulary. An inverted index stores only the non-zero entries. This makes retrieval fast even over millions of documents. Hybrid search combines BM25's lexical precision with dense vector search's semantic grasp. This page scores a set of documents against a query, shows the IDF and TF-saturation parts, and covers the k1/b parameter tradeoff.
The formula
score(D,Q) = Σt∈Q IDF(t) · f(t,D)(k1+1) / (f(t,D) + k1(1 − b + b·|D|/avgdl))
Three ideas, one formula. IDF(t) — a term that appears in fewer documents is more informative. So it is weighted higher. f(t,D) — more instances of a query term help, but with shrinking returns, controlled by k1. |D|/avgdl — a document's raw term count is normalised against its own length, controlled by b. So a long document does not win purely by holding more words.
Scoring by term overlap, carefully
BM25 is the keyword retrieval function that most search engines are built on. It scores a document against a query by adding up a share per matching term:
score(d, q) = Σt ∈ q IDF(t) × (f(t,d) × (k₁+1)) / (f(t,d) + k₁ × (1 − b + b × |d|/avgdl))
That looks worse than it is. Three ideas are encoded in it, and each fixes a set defect of naive term counting.
Rare terms count more (IDF). A match on "encephalopathy" is far more informative than a match on "the". The inverse document count term weights each match by how rare the term is across the collection.
Repetition saturates. A document mentioning a term twenty times is not twenty times more apt than one mentioning it once. The k₁ parameter controls how quickly the share flattens — often around 1.2. This means the third and fourth instances add much less than the first.
Long documents are penalised. A 10,000-word document will hold most query terms by chance. Dividing by document length next to the average corrects for it, with b (often 0.75) controlling how strongly.
Those three corrections are the entire gap between BM25 and TF-IDF. And they are why BM25 has been the standard for thirty years.
Where it beats embeddings
Sparse retrieval matches tokens, which is just what embeddings are bad at:
| Query | BM25 | Dense |
|---|---|---|
| "error code E4471" | Exact match | Blurs into other error codes |
| "section 12.3(b)" | Exact match | Approximate |
| "Nakamura et al 2019" | Exact match | Similar-looking citations |
| "holiday allowance" | Misses "annual leave" | Finds it |
| "cheapest shipping" | Misses "lowest-cost delivery" | Finds it |
The pattern: identifiers, names, codes and rare technical terms belong to BM25; paraphrase belongs to embeddings. Their failures do not overlap. This is the argument for using both.
Three further gains that matter operationally. BM25 needs no model, no GPU and no embedding step. So indexing is fast and cheap. It handles a brand-new term the moment it is indexed. Where an embedding model has never seen it. And it is transparent — you can explain just why a document scored highly. This matters for debugging and for audit.
Building an index
Sparse retrieval is an inverted index: for each term, the list of documents holding it and their term counts. A query touches only the lists for its own terms. This is why it is fast even over very large collections.
Preprocessing matters more here than for dense retrieval, because matching is literal:
- Lowercase, so "Password" matches "password".
- Tokenise always between indexing and querying — a mismatch means nothing matches.
- Remove stop words, or accept that IDF already suppresses them.
- Stem or lemmatise so "running" matches "run". Apply the same treatment to the query.
- Keep identifiers intact. A tokeniser that splits "E4471" into "E" and "4471" has destroyed the thing BM25 was best at.
That last point is the one that goes wrong most often in practice. And it is worth testing by name with a few known identifiers.
The two knobs TF-IDF was missing
BM25 is TF-IDF with two corrections -- term count saturates, and document length is discounted by a tunable amount. This implements it beside TF-IDF on the same documents so you can see just which ranking each correction changes.
Building on the same setup, the next block shows the left cases.
The below block covers the left steps.
The next block goes on with the left demonstrations.
Building on the same setup, the next block shows the left cases.
The below block covers the left steps.
The next block goes on with the left demonstrations.
Building on the same setup, the next block shows the left cases.
Things to try
- Read the default ranking. D2 leads — it repeats "list" four times and mentions each query term at least twice. And that repetition still wins even after the default length normalization is applied.
- Drag b to 0. Length normalization is off fully. D2's lead grows further still, since its extra length now costs it nothing at all.
- Push b to 1. Full length normalization. D2's lead shrinks — but does not disappear. A four-times repeated rare term is still worth more than one clean mention, even once length is fully accounted for.
- Now drag k1 to 0. This is the setting that really flips it: D1 overtakes D2. With k1 at 0, repeating a term buys nothing at all — only whether a term is present matters — so D2's four mentions of "list" count just the same as D1's one, and D1 wins on being the tighter, fully on-topic match.
- The lesson in that flip: b alone tempers term-stuffing; k1 is what really caps it. The two parameters are doing different jobs, and neither one alone fully neutralises a document that just repeats the query terms.
Why it still matters next to embeddings
BM25 gets exact terms right where embeddings can blur them — product codes, error messages, names, acronyms. It is also fully interpretable: each score decomposes into per-term shares. This is why the breakdown panel above can show just where a score came from. This is the sparse half of the hybrid search that most production RAG systems really run.
What to recall
BM25 scores a document by summing, over each query term it holds, that term's rarity across the corpus times a saturating function of how often it appears, normalised by document length. b controls how much long documents are penalised for being long; k1 controls how much repeated terms are rewarded before shrinking returns kick in. No embeddings, no training — and still the metric most search engines reach for first.
In code
from rank_bm25 import BM25Okapi
corpus = [doc.lower().split() for doc in documents] # consistent tokenising
bm25 = BM25Okapi(corpus, k1=1.5, b=0.75)
scores = bm25.get_scores("annual leave policy".lower().split())
top = sorted(range(len(scores)), key=lambda i: -scores[i])[:10]For anything beyond a prototype, use a real search engine — Elasticsearch, OpenSearch, Lucene, Tantivy or a vector database with built-in sparse support. They provide the inverted index, incremental updates, filtering, and analysers that handle identifiers and multiple languages properly.
Parameter guidance: k₁ between 1.2 and 2.0, b around 0.75. Neither is sensitive, and the defaults are almost always fine. Effort spent on tokenisation and analysers pays far better than tuning these.
Learned sparse retrieval
A newer family keeps the inverted-index machinery and learns the weights.
SPLADE uses a transformer to predict, for each document, a sparse weight over the whole vocabulary. Including terms that do not literally appear. A document about "annual leave" can be given weight on "holiday". So the sparse index gains some semantic matching.
The result sits between the two worlds: it retains the exact-match strength and index speed of sparse retrieval while handling some paraphrase. The cost is a model pass per document at indexing time, and larger posting lists.
Whether it beats BM25-plus-dense-hybrid depends on the corpus, and it is worth evaluating rather than assuming.
Questions people ask
Is BM25 outdated? No. It remains a strong baseline that dense retrieval alone often fails to beat, especially on technical corpora.
Do I need both BM25 and embeddings? For most real corpora, yes. Hybrid retrieval is the standard recommendation and often the largest single quality gain.
What is the gap from TF-IDF? Term-count saturation and document-length normalisation. Both matter, and both make BM25 better.
Should I remove stop words? IDF already suppresses them, so it is optional. Removing them shrinks the index.
Does it work for other languages? Yes, with right tokenisation and stemming. Languages without whitespace word boundaries need a proper analyser.
How do I combine it with dense scores? Reciprocal rank fusion. This uses ranks only and needs no score normalisation.
Recap in one screen
- BM25 scores term overlap with three corrections: rare terms count more, repetition saturates, long documents are penalised.
- It excels at identifiers, codes, names and rare terms. Just where embeddings blur.
- It needs no model or GPU, indexes instantly, and is fully explainable.
- Tokenisation must match between index and query, and must not split identifiers.
- Combine with dense retrieval via reciprocal rank fusion; learned sparse models such as SPLADE blur the boundary.