Course contentsShow
← Machine Learning and Deep Learning
Lesson 2342 of 3,53851. Recommender SystemsFree lesson

TF-IDF for Text-Based Items

Using TF-IDF to convert text descriptions into weighted feature vectors for content-based matching.

TF-IDF for Text-Based Items

What you'll learn: How to transform text descriptions into weighted feature vectors using TF-IDF for effective content-based matching in recommender systems.

The Core Idea

When building content-based recommender systems for items with text descriptions (movies, books, articles), you need numerical representations to compare items mathematically. TF-IDF (Term Frequency-Inverse Document Frequency) converts text into weighted vectors where important, distinctive words get higher scores than common ones.

Think of it like highlighting: if every movie description mentions "film," that word tells you nothing distinctive. But if only one thriller mentions "heist," that word becomes a strong identifying feature.

How TF-IDF Works

TF (Term Frequency) measures how often a word appears in a document:

  • More occurrences = potentially more important to that item

IDF (Inverse Document Frequency) downweights words that appear in many documents:

  • Common across all items = less discriminative power
  • Rare across the collection = more distinctive

Combined: TF-IDF = TF × IDF creates a balance where words that are frequent in one item but rare overall get the highest weights.

From Text to Vectors

Each item becomes a vector where each dimension represents a word from your vocabulary, weighted by its TF-IDF score. Items with similar content will have similar vectors (high cosine similarity), enabling you to recommend items based on content overlap rather than requiring collaborative signals.

For example, two sci-fi novels sharing terms like "starship," "alien," and "colony" will have overlapping high-weight features, making them mathematically close in vector space.

Key Takeaway: TF-IDF transforms text descriptions into numerical feature vectors by weighting words based on their importance within each item and distinctiveness across the entire collection, enabling content-based similarity calculations.