Negative Sampling in Word2Vec
You'll learn how Word2Vec becomes computationally practical by cleverly avoiding expensive calculations across the entire vocabulary.
The Computational Bottleneck
When training Word2Vec (either Skip-gram or CBOW), the naive approach requires computing a softmax over your entire vocabulary for every training example. If you have 100,000 words, that means calculating probabilities for all 100,000 words, normalizing them, and computing gradients—extremely expensive!
For a Skip-gram model predicting context words from a center word, the standard objective involves:
P(context|center) = exp(score) / Σ(exp(score) for all words)
That denominator is the killer—summing over every single word in your vocabulary.
The Negative Sampling Solution
Negative sampling transforms the problem from multi-class classification (which word is correct among 100,000?) into binary classification (is this word-pair real or fake?).
Instead of computing probabilities over all words, you:
- Take your positive example (actual context-center pair)
- Sample a small number (k=5-20) of "negative" words randomly from the vocabulary
- Train the model to distinguish the real pair from the fake pairs
Analogy: Instead of a teacher testing you on identifying the correct answer among 100,000 options, they show you 1 correct answer and 5-10 wrong ones. Much easier and faster!
The Math
For each positive pair, you maximize:
log σ(score_positive) + Σ log σ(-score_negative_i)
Where σ is the sigmoid function and you sum over k negative samples.
Key Takeaway: Negative sampling makes Word2Vec training feasible by replacing expensive softmax over the full vocabulary with simple binary classifications against a handful of randomly sampled negative examples.