ELMo: Deep Contextualized Word Representations
What you'll learn: How ELMo creates word embeddings that change based on context by combining forward and backward language models using bidirectional LSTMs.
The Core Problem ELMo Solves
Remember how static embeddings like Word2Vec give the word "bank" the same vector whether you're talking about a river bank or a financial bank? ELMo fixes this by reading the entire sentence in both directions before deciding what "bank" means in that specific context.
How ELMo Works
ELMo uses two separate LSTM language models:
- Forward LSTM: Reads the sentence left-to-right, predicting each next word
- Backward LSTM: Reads the sentence right-to-left, predicting each previous word
These are trained independently as language models, then their hidden states are combined.
Creating Context-Aware Embeddings
For each word in a sentence, ELMo:
- Runs the forward LSTM up to that word (capturing left context)
- Runs the backward LSTM from the end back to that word (capturing right context)
- Combines the hidden states from both directions at multiple layers
- Creates a weighted combination of these representations
Analogy: Imagine understanding a word by reading everything before it AND everything after it, like solving a crossword puzzle where you have clues from both horizontal and vertical directions.
Why Bidirectional Matters
The forward model knows "The bank was..." but hasn't seen "...near the river" yet. The backward model has that information. Together, they know the full context, allowing "bank" to get a different embedding than in "The bank approved my loan."
Key Takeaway: ELMo uses bidirectional LSTMs (one forward, one backward) to create embeddings that adapt to context by combining information from both directions of the sentence, solving the polysemy problem that plagued static embeddings.