Word2Vec: Continuous Bag of Words (CBOW)
What you'll learn: How CBOW predicts a target word by looking at its surrounding context, flipping the Skip-gram approach.
The Core Idea
While Skip-gram predicts context words from a center word, CBOW does the opposite: it predicts the center word from its surrounding context. Think of it like a fill-in-the-blank exercise: given "The cat sat on the ___", CBOW learns to predict "mat" by averaging information from all the context words.
How CBOW Works
CBOW takes multiple context words as input (say, 2 words before and 2 words after the target) and combines their embeddings—typically by averaging—to predict the center word. This makes CBOW faster than Skip-gram because it processes multiple context words in a single forward pass rather than making separate predictions for each context word.
The Architecture Flow
- Input: Context word indices (e.g., ["cat", "sat", "on", "the"])
- Lookup: Retrieve embedding vectors for each context word
- Combine: Average (or sum) these embeddings into a single vector
- Predict: Pass through a classifier to predict the center word ("mat")
- Learn: Update embeddings based on prediction error
CBOW vs Skip-gram
CBOW is generally faster and works better with frequent words because it smooths over distributional information by averaging context. Skip-gram, conversely, treats each context word separately and often performs better with rare words and smaller datasets.
Key Takeaway: CBOW learns word embeddings by predicting a target word from its context through averaged context vectors, offering a computationally efficient alternative to Skip-gram that excels with frequent words in large datasets.