Course contentsShow
← Machine Learning and Deep Learning
Lesson 1136 of 3,53825. Word Embeddings and Contextual RepresentationsFree lesson

From RNNs to Transformers for Contextualization

Why Transformer-based contextual embeddings replaced LSTM-based approaches like ELMo for efficiency and quality.

From RNNs to Transformers for Contextualization

You'll learn why Transformer-based contextual embeddings superseded LSTM-based approaches like ELMo.

The Evolution of Contextualization

ELMo revolutionized NLP by using bidirectional LSTMs to create contextualized word embeddings. However, despite its success, ELMo had inherent limitations stemming from its RNN architecture: sequential processing bottlenecks, difficulty capturing very long-range dependencies, and computational inefficiency during training.

Why Transformers Won

Transformers solved these problems fundamentally:

Parallelization: Unlike RNNs that process tokens one-by-one, Transformers process entire sequences simultaneously using self-attention. This makes training orders of magnitude faster on modern GPUs.

Better Long-Range Dependencies: Attention creates direct connections between any two tokens in constant computational steps (one attention layer), whereas RNNs must propagate information through many sequential steps, causing gradient degradation.

Quality of Representations: Self-attention explicitly models relationships between all token pairs, allowing richer contextual understanding. ELMo's LSTMs could only look forward or backward sequentially, limiting their ability to integrate distant context.

Scalability: Transformers follow better scaling laws—adding more parameters and data consistently improves performance, which enabled models like BERT, GPT, and beyond.

The Trade-off

Transformers require more memory (quadratic in sequence length) and more data to train effectively. However, for most modern NLP tasks, the efficiency gains and quality improvements far outweigh these costs, which is why virtually all state-of-the-art contextual embedding models today (BERT, RoBERTa, GPT-3) are Transformer-based.

Key Takeaway: Transformers replaced LSTM-based contextualization methods like ELMo because they enable parallel training, handle long-range dependencies more effectively through direct attention connections, and produce higher-quality representations that scale better with data and parameters.