Bag-of-Words and TF-IDF Features
You'll learn how to convert text into numerical features using two classical approaches that count and weight words for traditional machine learning.
What Are These Methods?
Before neural networks dominated NLP, we needed ways to turn text into numbers that traditional ML models (like logistic regression or SVMs) could process. These two approaches treat documents as collections of words, ignoring order but capturing content.
Bag-of-Words (BoW)
Imagine dumping all words from a document into a bag, losing their order but counting how many times each appears. That's BoW! Each document becomes a vector where each position represents a word from your vocabulary, and the value is how often that word appears.
Example: "The cat sat. The dog sat." becomes a vector like [2, 1, 1, 1] for vocabulary [the, cat, sat, dog].
TF-IDF (Term Frequency-Inverse Document Frequency)
BoW treats all words equally, but some words like "the" appear everywhere and carry little meaning. TF-IDF fixes this by weighting words:
- TF (Term Frequency): How often a word appears in this document
- IDF (Inverse Document Frequency): How rare the word is across all documents
Common words get low scores; rare, distinctive words get high scores. It's like highlighting the words that make each document unique.
Why Use These?
These features work well with traditional ML classifiers for text classification tasks. They're simple, interpretable, and surprisingly effective for many problems—though they can't capture word order or semantics like modern embeddings do.
Key Takeaway: Bag-of-Words counts word occurrences, while TF-IDF weights those counts by how distinctive each word is, turning text into numerical vectors for traditional machine learning models.