Course contentsShow
← Machine Learning and Deep Learning
Lesson 1049 of 3,53824. The Transformer ArchitectureFree lesson

The Limitations of RNNs that Attention Solves

Understand sequential bottlenecks, vanishing gradients, and why RNNs struggle with long-range dependencies.

The Limitations of RNNs that Attention Solves

What you'll learn: How RNNs fundamentally struggle with long sequences and why attention mechanisms are necessary to overcome these challenges.

The Core Problem

Even with LSTMs and GRUs helping gradient flow, RNNs face a sequential bottleneck: they must process tokens one-by-one, left-to-right. This creates three critical limitations:

1. Long-Range Dependencies

Despite gates, RNNs still struggle to connect information separated by many time steps. If a crucial word appears at position 1 and is needed at position 100, the signal must pass through 99 intermediate states—like a game of telephone where the message degrades.

2. The Vanishing Gradient Problem Persists

While LSTMs mitigate gradient vanishing, they don't eliminate it completely. The longer the sequence, the harder it becomes to propagate meaningful gradients backward through all those time steps.

3. Sequential Processing Bottleneck

RNNs can't parallelize within a sequence. You must compute hidden state h₁ before h₂, h₂ before h₃, and so on. This makes training painfully slow on modern GPUs designed for parallel computation.

How Attention Helps

Attention allows the model to directly access any position in the sequence without passing through intermediate states. It's like having random-access memory instead of tape storage—you can jump straight to what you need.

By computing attention scores between all positions simultaneously, attention:

  • Eliminates the sequential bottleneck (all positions computed in parallel)
  • Provides direct paths for gradient flow (no long chain of dependencies)
  • Lets the model focus on relevant context regardless of distance

Key Takeaway: RNNs force information through sequential bottlenecks that cause gradient problems and prevent parallelization; attention provides direct, parallelizable connections between any positions in a sequence, solving all three limitations at once.