Limitations of Static Embeddings
What You'll Learn
You'll understand why traditional word embeddings like Word2Vec and GloVe have a critical flaw: they assign one fixed vector per word, regardless of how that word is used.
The One-Representation Problem
Static embeddings (Word2Vec, GloVe) create a single, unchanging vector for each word in your vocabulary. This means "bank" gets the exact same representation whether you're talking about a river bank or a financial institution. The word "apple" has identical embeddings when discussing fruit or technology companies.
Why This Matters
Think of it like having one photograph to represent a person in every context of their life—at work, at home, playing sports, or sleeping. That single image can't capture all the different roles and contexts where that person appears.
In natural language:
- Polysemy: Words have multiple meanings ("bat" = animal or sports equipment)
- Context-dependent usage: "The movie was sick" vs "I feel sick" use the same embedding despite opposite sentiments
- Named entities: "Paris" (city) vs "Paris" (person's name) are indistinguishable
The Technical Reality
When you load Word2Vec or GloVe embeddings, you get a lookup table: each word maps to one fixed 300-dimensional vector (or whatever dimension you chose). No matter what sentence you place that word in, it retrieves the same vector. The embedding has no awareness of surrounding words, grammar, or semantic context.
This limitation motivated the development of contextual embeddings where representations change based on usage—but that's for future lessons!
Key Takeaway: Static embeddings give every word exactly one fixed representation, making them unable to distinguish between different meanings, contexts, or uses of the same word—a fundamental limitation for understanding natural language.