Contextual vs Static Embeddings
What you'll learn: How embeddings can either stay the same for a word or change based on surrounding context.
The Core Difference
Until now, we've talked about embeddings as if each word or token always maps to the same vector. That's how static embeddings like Word2Vec and GloVe work—the word "bank" always gets the exact same vector, whether you're talking about a river bank or a financial bank.
Contextual embeddings (like those from BERT and modern transformers) generate different vectors for the same word depending on the sentence it appears in. The model reads the whole sentence and adjusts each word's embedding based on its neighbors.
A Simple Analogy
Think of static embeddings like a dictionary definition: "bank" has one fixed meaning you look up every time.
Contextual embeddings are like reading the word in a full sentence—you understand "I sat by the river bank" differently than "I went to the bank for a loan" because you see the context around it.
Why This Matters
Static embeddings struggle with:
- Polysemy: Words with multiple meanings
- Context-dependent nuances: "good" in "good food" vs "good enough"
Contextual embeddings capture:
- Meaning in context: Same word, different vectors for different uses
- Relationships to nearby words: The embedding changes based on what's around it
Modern AI systems (BERT, GPT, etc.) use contextual embeddings because they're dramatically better at understanding what words actually mean in each specific situation.
Key Takeaway: Static embeddings give the same vector to a word every time; contextual embeddings generate different vectors based on the surrounding sentence, capturing meaning more accurately.