Course contentsShow
← Machine Learning and Deep Learning
Lesson 1133 of 3,53825. Word Embeddings and Contextual RepresentationsFree lesson

ELMo: Deep Contextualized Word Representations

Using bidirectional LSTMs to create context-aware embeddings by combining forward and backward language models.

ELMo: Deep Contextualized Word Representations

What you'll learn: How ELMo creates word embeddings that change based on context by combining forward and backward language models using bidirectional LSTMs.

The Core Problem ELMo Solves

Remember how static embeddings like Word2Vec give the word "bank" the same vector whether you're talking about a river bank or a financial bank? ELMo fixes this by reading the entire sentence in both directions before deciding what "bank" means in that specific context.

How ELMo Works

ELMo uses two separate LSTM language models:

  1. Forward LSTM: Reads the sentence left-to-right, predicting each next word
  2. Backward LSTM: Reads the sentence right-to-left, predicting each previous word

These are trained independently as language models, then their hidden states are combined.

Creating Context-Aware Embeddings

For each word in a sentence, ELMo:

  • Runs the forward LSTM up to that word (capturing left context)
  • Runs the backward LSTM from the end back to that word (capturing right context)
  • Combines the hidden states from both directions at multiple layers
  • Creates a weighted combination of these representations

Analogy: Imagine understanding a word by reading everything before it AND everything after it, like solving a crossword puzzle where you have clues from both horizontal and vertical directions.

Why Bidirectional Matters

The forward model knows "The bank was..." but hasn't seen "...near the river" yet. The backward model has that information. Together, they know the full context, allowing "bank" to get a different embedding than in "The bank approved my loan."

Key Takeaway: ELMo uses bidirectional LSTMs (one forward, one backward) to create embeddings that adapt to context by combining information from both directions of the sentence, solving the polysemy problem that plagued static embeddings.