Course contentsShow
← System Design
Lesson 1747 of 1,91942. Real-World System: Search and TypeaheadFree lesson

TF-IDF: Term Frequency-Inverse Document Frequency

Calculating term importance by balancing frequency within a document against corpus-wide rarity.

TF-IDF: Term Frequency-Inverse Document Frequency

What You'll Learn

You'll learn how TF-IDF balances a word's frequency within a document against its rarity across all documents to measure its true importance for search ranking.

The Core Problem

Not all words in a document matter equally. The word "the" might appear 50 times in a document, but it tells you nothing about what the document is about. Meanwhile, "quantum" appearing just 3 times might be extremely significant—if most other documents never mention it.

TF-IDF solves this by combining two metrics:

Term Frequency (TF)

How often does the term appear in this specific document?

A term that appears frequently in a document is likely important to that document. You might count raw occurrences or normalize by document length:

TF = (occurrences of term in doc) / (total terms in doc)

Inverse Document Frequency (IDF)

How rare is this term across the entire collection?

If a term appears in every document, it's not useful for distinguishing documents. Rare terms are more valuable:

IDF = log(total documents / documents containing term)

The logarithm dampens the effect so extremely rare words don't dominate unfairly.

The TF-IDF Score

Multiply them together:

TF-IDF = TF × IDF

This gives you a score that's high when a term appears frequently in a document and rarely across the corpus.

Real-World Analogy

Think of restaurant reviews. If a review mentions "delicious" (appears in 90% of reviews), that's not distinctive. But if it mentions "molecular gastronomy" (appears in 2% of reviews) multiple times, that review is clearly about a specific type of restaurant—and should rank highly for that search term.

Key Takeaway: TF-IDF scores balance local term frequency with global rarity, ensuring search results surface documents where query terms are both prominent and distinctive.