Course contentsShow
← Machine Learning and Deep Learning
Lesson 1134 of 3,53825. Word Embeddings and Contextual RepresentationsFree lesson

ELMo Architecture and Pretraining

How ELMo stacks biLSTMs, trains on language modeling, and extracts layer-wise representations.

ELMo Architecture and Pretraining

You'll learn: How ELMo builds deep contextualized representations by stacking bidirectional LSTMs trained on language modeling tasks.

The Architecture: Stacked BiLSTMs

ELMo (Embeddings from Language Models) uses a multi-layer bidirectional LSTM architecture. Think of it like a sandwich: a forward LSTM reads the sentence left-to-right, while a backward LSTM reads right-to-left at the same time. These aren't just one layer—ELMo stacks multiple biLSTM layers on top of each other (typically 2-3 layers).

The input starts with standard token embeddings (like Word2Vec or random initialization), then each biLSTM layer processes the sequence in both directions independently, creating richer representations at each level.

Pretraining: Language Modeling Objective

ELMo is pretrained on a massive text corpus using language modeling—predicting the next word given previous context. Specifically:

  • The forward LSTM predicts the next token given all previous tokens
  • The backward LSTM predicts the previous token given all future tokens
  • Both objectives are trained simultaneously

This is like learning to fill in blanks from both directions: "The cat sat on the " (forward) and " cat sat on the mat" (backward).

Layer-Wise Representations

Here's the key innovation: ELMo doesn't just use the final layer. It extracts representations from all layers and combines them. Lower layers capture syntax (like parts of speech), while higher layers capture semantics (like word meaning). Each task can learn its own weighted combination of these layer outputs.

The pretraining phase learns general language patterns. Later, when you use ELMo embeddings in your specific task, you can fine-tune how to weight each layer's contribution.

Key Takeaway: ELMo pretrains stacked biLSTMs with bidirectional language modeling objectives, then extracts and combines representations from all layers to create deep, context-sensitive word embeddings.