BM25 — AI Glossary

Reference details

Related terms

• Hybrid Search
• Semantic Search

How it works

1. Documents and the query are reduced to terms, so matching is on exact tokens rather than on meaning.
2. Each term is weighted by inverse document frequency: a term appearing in few documents carries more signal than one appearing everywhere.
3. Term frequency within a document is saturated rather than counted linearly, controlled by k1 — the tenth occurrence of a word adds far less than the second.
4. Document length is normalised by b, so a long document does not score highly merely for containing the query terms somewhere.
5. Scores are summed across query terms, and an inverted index makes the lookup fast without comparing the query against every document.

Example: Saturation is why term stuffing stops working

With k1 = 1.2 and a term appearing f times, the frequency component is f / (f + k1). One occurrence gives 1/2.2 ≈ 0.45; five give 5/6.2 ≈ 0.81; fifty give 50/51.2 ≈ 0.98. Going from one to five occurrences adds about 0.36, while going from five to fifty adds about 0.17 for ten times the repetition. The curve is the defence: repeating a keyword cannot dominate the score, which is a property an embedding similarity does not have in the same explicit form.

Commonly confused with

A specific ranking function; sparse retrieval is the family it belongs to. It scores on term overlap weighted by rarity and document length, so it matches exact strings — product codes, error messages, surnames — that embeddings blur together. It cannot match a paraphrase, which is the complementary weakness that makes hybrid search worth the extra index.

When to use it

Reach for it when:

• Queries contain exact strings that must match — identifiers, error codes, part numbers, surnames.
• As the lexical half of a hybrid stack, where it covers the exact matches dense retrieval misses.
• A strong baseline is needed before any embedding infrastructure exists.

Reach for something else when:

• Queries and documents use different vocabulary for the same concept, where term overlap is absent and dense retrieval wins.
• The corpus is multilingual and queries cross languages, with no shared terms to match.
• As the sole retriever for natural-language questions over prose, where paraphrase is the norm.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to BM25 — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path