What is BM25 Ranking in Information Search?

BM25 (often called Okapi BM25) is the benchmark probabilistic ranking function used by modern search engines to evaluate the relevance of documents to a given search query.

Developed by Stephen E. Robertson, Karen Spärck Jones, and colleagues at City University London in the 1990s, BM25 replaced primitive keyword-matching heuristics with a mathematically principled model that remains the standard baseline in information retrieval.

The mathematical mechanics of BM25

Given a query $Q$ containing keywords $q_1, q_2, \dots, q_n$, the BM25 score of a document $D$ is calculated as:

$$\text{Score}(D, Q) = \sum_{i=1}^n \text{IDF}(q_i) \cdot \frac{f(q_i, D) \cdot (k_1 + 1)}{f(q_i, D) + k_1 \cdot \left(1 - b + b \cdot \frac{|D|}{\text{avgdl}}\right)}$$

Where:

• $f(q_i, D)$ is the term frequency of keyword $q_i$ within document $D$.
• $|D|$ is the length of document $D$ in words.
• $\text{avgdl}$ is the average document length across the entire collection.
• $k_1$ is a tuning parameter (typically 1.2 to 2.0) controlling term frequency saturation.
• $b$ is a tuning parameter (typically 0.75) controlling document length normalization.

Two critical innovations of BM25

1. Term frequency saturation ($k_1$)

In basic TF-IDF, a note that mentions “encryption” 100 times scores 100 times higher than a note that mentions it once. In reality, a note with 5 mentions is usually just as relevant as one with 100. BM25 enforces an asymptotic curve: as term frequency increases, additional occurrences provide diminishing marginal score gains.

2. Document length penalization ($b$)

A 50,000-word engineering manual will naturally contain more query keywords than a concise 300-word meeting summary. BM25 normalizes by document length, penalizing long documents that merely contain keywords by chance while boosting compact, high-density notes.

BM25 in Oyma on macOS

In Oyma, BM25 is built directly into our local SQLite FTS5 search pipeline. When searching across hundreds of meeting notes, customer interviews, and architectural specifications, Oyma uses BM25 to ensure that concise, highly focused notes surface ahead of rambling transcripts, returning relevant insights in milliseconds.