Course contentsShow
← Machine Learning and Deep Learning
Lesson 1998 of 3,53843. Retrieval-Augmented Generation: Advanced TechniquesFree lesson

Keyword Search Fundamentals: BM25

How BM25 scoring works for term frequency and inverse document frequency in lexical retrieval.

Keyword Search Fundamentals: BM25

What you'll learn: How BM25 scores documents using term frequency and inverse document frequency to complement vector-based retrieval in RAG systems.

The Problem with Pure Vector Search

Vector embeddings capture semantic meaning but sometimes miss exact keyword matches. If a user searches for "Python 3.11 release date," semantic search might return general Python documentation, while a keyword-based approach would prioritize documents containing those exact terms.

What Is BM25?

BM25 (Best Match 25) is a probabilistic ranking function used in traditional information retrieval. It scores documents based on how relevant they are to a query by analyzing term frequency (how often query words appear in a document) and inverse document frequency (how rare those terms are across all documents).

How BM25 Scoring Works

BM25 calculates a relevance score for each document using three key factors:

  1. Term Frequency (TF): Documents mentioning query terms more often score higher, but with diminishing returns (mentioning "Python" 100 times isn't 100x better than 10 times)
  2. Inverse Document Frequency (IDF): Rare terms like "BM25" are weighted more heavily than common words like "the"
  3. Document Length Normalization: Longer documents are penalized to prevent them from unfairly dominating results

The algorithm balances these factors with tunable parameters (typically k1 for term saturation and b for length normalization) to produce a final relevance score.

Why BM25 Matters in RAG

In hybrid retrieval systems, BM25 excels at exact-match queries, technical terms, and entity names—complementing the semantic understanding of dense embeddings. This lexical approach ensures you don't lose precision when keywords matter.

Key Takeaway: BM25 uses term frequency and inverse document frequency with length normalization to provide keyword-based scoring that captures exact matches vector search might miss.