TF-IDF: Term Frequency-Inverse Document Frequency
What You'll Learn
You'll learn how TF-IDF balances a word's frequency within a document against its rarity across all documents to measure its true importance for search ranking.
The Core Problem
Not all words in a document matter equally. The word "the" might appear 50 times in a document, but it tells you nothing about what the document is about. Meanwhile, "quantum" appearing just 3 times might be extremely significant—if most other documents never mention it.
TF-IDF solves this by combining two metrics:
Term Frequency (TF)
How often does the term appear in this specific document?
A term that appears frequently in a document is likely important to that document. You might count raw occurrences or normalize by document length:
TF = (occurrences of term in doc) / (total terms in doc)
Inverse Document Frequency (IDF)
How rare is this term across the entire collection?
If a term appears in every document, it's not useful for distinguishing documents. Rare terms are more valuable:
IDF = log(total documents / documents containing term)
The logarithm dampens the effect so extremely rare words don't dominate unfairly.
The TF-IDF Score
Multiply them together:
TF-IDF = TF × IDF
This gives you a score that's high when a term appears frequently in a document and rarely across the corpus.
Real-World Analogy
Think of restaurant reviews. If a review mentions "delicious" (appears in 90% of reviews), that's not distinctive. But if it mentions "molecular gastronomy" (appears in 2% of reviews) multiple times, that review is clearly about a specific type of restaurant—and should rank highly for that search term.
Key Takeaway: TF-IDF scores balance local term frequency with global rarity, ensuring search results surface documents where query terms are both prominent and distinctive.