Course contentsShow
← System Design
Lesson 1740 of 1,91942. Real-World System: Search and TypeaheadFree lesson

TF-IDF Scoring Fundamentals

Using term frequency and inverse document frequency to score document relevance for ranking search results.

TF-IDF Scoring Fundamentals

What You'll Learn

You'll learn how search engines use TF-IDF (Term Frequency-Inverse Document Frequency) to calculate relevance scores that determine which documents appear first in search results.

Why Relevance Scoring Matters

After your search engine has used the inverted index to find all documents containing your query terms, it needs to decide which ones are most relevant. Not all matches are equal—a document mentioning "python" once among thousands of words is less relevant than one focused specifically on Python programming.

The Two Components

Term Frequency (TF): How often does a query term appear in a specific document? The more frequently a term appears, the more relevant that document likely is for that term.

Analogy: If you're searching for "chocolate cake recipe" and one document mentions "chocolate" 15 times while another mentions it once, the first is probably more about chocolate.

Inverse Document Frequency (IDF): How rare is this term across all documents? Common words like "the" or "is" appear everywhere and don't help distinguish relevance. Rare, specific terms matter more.

Analogy: Everyone says "hello," but few discuss "kubernetes"—so "kubernetes" is more meaningful for ranking.

The Formula

TF-IDF multiplies these together:

TF-IDF Score = (Term Frequency) × (Inverse Document Frequency)

High TF means the term appears often in this document. High IDF means the term is rare globally, making it distinctive. Together, they boost documents that frequently mention rare, meaningful terms.

In Practice

When a user searches "distributed systems architecture," the search engine calculates TF-IDF scores for each term in each matching document, then combines them to produce a final relevance score used for ranking.

Key Takeaway: TF-IDF scores rank documents by balancing how often query terms appear in each document (TF) against how distinctive those terms are globally (IDF), surfacing the most relevant results first.