Course contentsShow
← Machine Learning and Deep Learning
Lesson 1121 of 3,53825. Word Embeddings and Contextual RepresentationsFree lesson

Negative Sampling in Word2Vec

Understand how negative sampling makes Word2Vec training efficient by avoiding expensive softmax over full vocabulary.

Negative Sampling in Word2Vec

You'll learn how Word2Vec becomes computationally practical by cleverly avoiding expensive calculations across the entire vocabulary.

The Computational Bottleneck

When training Word2Vec (either Skip-gram or CBOW), the naive approach requires computing a softmax over your entire vocabulary for every training example. If you have 100,000 words, that means calculating probabilities for all 100,000 words, normalizing them, and computing gradients—extremely expensive!

For a Skip-gram model predicting context words from a center word, the standard objective involves:

P(context|center) = exp(score) / Σ(exp(score) for all words)

That denominator is the killer—summing over every single word in your vocabulary.

The Negative Sampling Solution

Negative sampling transforms the problem from multi-class classification (which word is correct among 100,000?) into binary classification (is this word-pair real or fake?).

Instead of computing probabilities over all words, you:

  1. Take your positive example (actual context-center pair)
  2. Sample a small number (k=5-20) of "negative" words randomly from the vocabulary
  3. Train the model to distinguish the real pair from the fake pairs

Analogy: Instead of a teacher testing you on identifying the correct answer among 100,000 options, they show you 1 correct answer and 5-10 wrong ones. Much easier and faster!

The Math

For each positive pair, you maximize:

log σ(score_positive) + Σ log σ(-score_negative_i)

Where σ is the sigmoid function and you sum over k negative samples.

Key Takeaway: Negative sampling makes Word2Vec training feasible by replacing expensive softmax over the full vocabulary with simple binary classifications against a handful of randomly sampled negative examples.