Course contentsShow
← Machine Learning and Deep Learning
Lesson 1120 of 3,53825. Word Embeddings and Contextual RepresentationsFree lesson

Word2Vec: Continuous Bag of Words (CBOW)

Learn the CBOW approach that predicts a center word from its surrounding context words.

Word2Vec: Continuous Bag of Words (CBOW)

What you'll learn: How CBOW predicts a target word by looking at its surrounding context, flipping the Skip-gram approach.

The Core Idea

While Skip-gram predicts context words from a center word, CBOW does the opposite: it predicts the center word from its surrounding context. Think of it like a fill-in-the-blank exercise: given "The cat sat on the ___", CBOW learns to predict "mat" by averaging information from all the context words.

How CBOW Works

CBOW takes multiple context words as input (say, 2 words before and 2 words after the target) and combines their embeddings—typically by averaging—to predict the center word. This makes CBOW faster than Skip-gram because it processes multiple context words in a single forward pass rather than making separate predictions for each context word.

The Architecture Flow

  1. Input: Context word indices (e.g., ["cat", "sat", "on", "the"])
  2. Lookup: Retrieve embedding vectors for each context word
  3. Combine: Average (or sum) these embeddings into a single vector
  4. Predict: Pass through a classifier to predict the center word ("mat")
  5. Learn: Update embeddings based on prediction error

CBOW vs Skip-gram

CBOW is generally faster and works better with frequent words because it smooths over distributional information by averaging context. Skip-gram, conversely, treats each context word separately and often performs better with rare words and smaller datasets.

Key Takeaway: CBOW learns word embeddings by predicting a target word from its context through averaged context vectors, offering a computationally efficient alternative to Skip-gram that excels with frequent words in large datasets.