BERT Embeddings as Contextual Representations
You'll learn how BERT creates embeddings that change based on surrounding context, unlike static word embeddings.
The Core Problem with Static Embeddings
Remember Word2Vec and GloVe? They give the word "bank" the exact same vector whether you're talking about a river bank or a financial bank. That's a fundamental limitation we've been building toward solving.
How BERT Solves This
BERT (Bidirectional Encoder Representations from Transformers) uses a bidirectional Transformer encoder to produce embeddings that depend on the entire sentence context.
Here's how it works:
1. Bidirectional Processing
Unlike ELMo's separate left-to-right and right-to-left LSTMs, BERT's self-attention mechanism looks at all words simultaneously in both directions. When processing "bank" in "I deposited money at the bank," BERT's attention layers examine "deposited" and "money" on the left and can see the full context at once.
2. Token-Level Context Integration
Each token passes through multiple Transformer encoder layers. At each layer:
- Self-attention computes relationships with every other token
- The representation gets refined based on these relationships
- By the final layer, each token's embedding captures its meaning in this specific sentence
3. Different Sentences, Different Embeddings
The same word produces different embeddings in different contexts:
- "The bank was steep" → embedding influenced by "steep"
- "The bank approved the loan" → embedding influenced by "approved" and "loan"
The Analogy
Think of static embeddings as a dictionary definition—always the same. BERT embeddings are like understanding a word through the conversation it appears in—the meaning shifts based on what's being discussed.
Key Takeaway: BERT's bidirectional Transformer encoder produces context-dependent embeddings by using self-attention to integrate information from all surrounding tokens, giving each word a unique representation for each sentence.