Keyword Search Fundamentals: BM25
What you'll learn: How BM25 scores documents using term frequency and inverse document frequency to complement vector-based retrieval in RAG systems.
The Problem with Pure Vector Search
Vector embeddings capture semantic meaning but sometimes miss exact keyword matches. If a user searches for "Python 3.11 release date," semantic search might return general Python documentation, while a keyword-based approach would prioritize documents containing those exact terms.
What Is BM25?
BM25 (Best Match 25) is a probabilistic ranking function used in traditional information retrieval. It scores documents based on how relevant they are to a query by analyzing term frequency (how often query words appear in a document) and inverse document frequency (how rare those terms are across all documents).
How BM25 Scoring Works
BM25 calculates a relevance score for each document using three key factors:
- Term Frequency (TF): Documents mentioning query terms more often score higher, but with diminishing returns (mentioning "Python" 100 times isn't 100x better than 10 times)
- Inverse Document Frequency (IDF): Rare terms like "BM25" are weighted more heavily than common words like "the"
- Document Length Normalization: Longer documents are penalized to prevent them from unfairly dominating results
The algorithm balances these factors with tunable parameters (typically k1 for term saturation and b for length normalization) to produce a final relevance score.
Why BM25 Matters in RAG
In hybrid retrieval systems, BM25 excels at exact-match queries, technical terms, and entity names—complementing the semantic understanding of dense embeddings. This lexical approach ensures you don't lose precision when keywords matter.
Key Takeaway: BM25 uses term frequency and inverse document frequency with length normalization to provide keyword-based scoring that captures exact matches vector search might miss.