Word2Vec: Skip-gram Architecture
You'll learn: How the skip-gram model learns word embeddings by predicting surrounding context words from a center word.
The Core Idea
While you've learned that the distributional hypothesis tells us "words that appear in similar contexts have similar meanings," the skip-gram architecture is one specific way to learn those dense word embeddings. Skip-gram flips the prediction task: given a single center word, predict which words are likely to appear nearby in its context window.
Think of it like this: if I say "coffee," you might immediately think of words like "hot," "drink," "morning," or "cup." Skip-gram trains a neural network to make exactly these predictions, and in doing so, learns meaningful vector representations.
How Skip-gram Works
The architecture is surprisingly simple:
- Input: A center word (encoded as one-hot vector)
- Hidden layer: Projects the word into a lower-dimensional embedding space (the weights here become your word vectors)
- Output: Predicts probability distributions for multiple context words (words within a window around the center word)
For example, in the sentence "The cat sat on the mat," if "sat" is your center word and window size is 2, you'd try to predict "The," "cat," "on," and "the" independently from "sat."
The magic happens because words that appear in similar contexts will need similar embedding vectors to make accurate predictions. "Dog" and "cat" both predict similar context words, so their learned vectors end up close together.
Key Takeaway: Skip-gram learns word embeddings by training a shallow neural network to predict context words from a center word, forcing similar words into similar vector spaces based on the distributional hypothesis.