Limitations of Static Word Embeddings
You'll learn: Why assigning a single vector to each word fails to capture the richness and context-dependency of human language.
The Core Problem
Static word embeddings like Word2Vec and GloVe assign one fixed vector to each word, regardless of how that word is used. This seems efficient, but language is far more nuanced than that.
Why Single Vectors Fall Short
Polysemy: One Word, Many Meanings
The word "bank" has multiple meanings:
- Financial institution: "I went to the bank to deposit money."
- River edge: "We sat by the bank of the river."
Static embeddings give both uses the same vector, averaging across all contexts in the training data. The model can't distinguish which meaning applies.
Context-Dependent Meaning
Even beyond polysemy, words shift meaning subtly:
- "The movie was sick" (awesome)
- "I feel sick" (ill)
A single vector cannot capture these contextual nuances.
Sentence-Level Semantics Lost
Static embeddings treat sentences as bags of word vectors. They miss:
- Word order: "Dog bites man" vs. "Man bites dog"
- Syntax and grammar: Relationships between words
- Negation: "not good" should mean something different than "good"
The Analogy
Imagine describing a person with one photograph taken years ago. That photo can't show whether they're happy today, what they're wearing now, or who they're with. Similarly, a static embedding is a "snapshot" that ignores the word's current context.
Key Takeaway: Static word embeddings fail because they assign one fixed representation per word, unable to capture polysemy, context-dependent meanings, or sentence-level relationships—motivating the need for contextual embeddings.