Comparing Contextual Embedding Approaches
What you'll learn: How bidirectional (BERT), unidirectional (GPT), and hybrid contextual embedding methods differ in how they capture word meaning.
The Core Difference: Direction of Context
When creating contextual embeddings, the fundamental choice is which surrounding words influence a token's representation.
Bidirectional (BERT-style)
BERT reads text in both directions simultaneously. When encoding the word "bank" in "I deposited money at the bank," BERT sees all words—both before ("deposited money") and after (if any). This gives the richest context but requires seeing the complete sequence upfront.
Analogy: Reading a sentence with both eyes open, seeing everything at once.
Unidirectional (GPT-style)
GPT reads text left-to-right only (causal/autoregressive). When encoding "bank," it only sees "I deposited money at the" but not what comes after. This is necessary for text generation (you can't see the future you're creating) but captures less context for understanding.
Analogy: Reading word-by-word with a sliding window that only reveals what you've already read.
Hybrid Approaches
Some models combine both:
- ELMo trains separate forward and backward LSTMs, then concatenates their representations
- Some architectures use bidirectional encoders with unidirectional decoders
When to Use Which
- BERT (bidirectional): Best for understanding tasks (classification, NER, QA) where you have the full input
- GPT (unidirectional): Required for generation tasks; also works for understanding by treating it as completion
- Hybrid (ELMo): Bridges both worlds but less powerful than transformer-based approaches
Key Takeaway: Bidirectional embeddings (BERT) see full context and excel at understanding; unidirectional embeddings (GPT) only see past context, enabling generation but potentially missing future clues; hybrid methods attempt to capture both directions separately.