TF-IDF Scoring Fundamentals
What You'll Learn
You'll learn how search engines use TF-IDF (Term Frequency-Inverse Document Frequency) to calculate relevance scores that determine which documents appear first in search results.
Why Relevance Scoring Matters
After your search engine has used the inverted index to find all documents containing your query terms, it needs to decide which ones are most relevant. Not all matches are equal—a document mentioning "python" once among thousands of words is less relevant than one focused specifically on Python programming.
The Two Components
Term Frequency (TF): How often does a query term appear in a specific document? The more frequently a term appears, the more relevant that document likely is for that term.
Analogy: If you're searching for "chocolate cake recipe" and one document mentions "chocolate" 15 times while another mentions it once, the first is probably more about chocolate.
Inverse Document Frequency (IDF): How rare is this term across all documents? Common words like "the" or "is" appear everywhere and don't help distinguish relevance. Rare, specific terms matter more.
Analogy: Everyone says "hello," but few discuss "kubernetes"—so "kubernetes" is more meaningful for ranking.
The Formula
TF-IDF multiplies these together:
TF-IDF Score = (Term Frequency) × (Inverse Document Frequency)
High TF means the term appears often in this document. High IDF means the term is rare globally, making it distinctive. Together, they boost documents that frequently mention rare, meaningful terms.
In Practice
When a user searches "distributed systems architecture," the search engine calculates TF-IDF scores for each term in each matching document, then combines them to produce a final relevance score used for ranking.
Key Takeaway: TF-IDF scores rank documents by balancing how often query terms appear in each document (TF) against how distinctive those terms are globally (IDF), surfacing the most relevant results first.