The Vanishing Gradient Problem in RNNs
You'll learn: Why RNNs struggle to learn from events that happened many time steps ago, even though they're designed for sequential data.
The Core Problem
When training RNNs with Backpropagation Through Time (BPTT), gradients must flow backward through many time steps to update early hidden states. At each time step, the gradient gets multiplied by the recurrent weight matrix and activation function derivatives. If these values are less than 1, repeated multiplication causes the gradient to shrink exponentially—it "vanishes" before reaching distant past states.
Think of it like a whisper game: Each person passes a message to the next, but speaks slightly quieter each time. After 50 people, the original message is inaudible. Similarly, after many time steps, the gradient signal from the loss becomes too weak to effectively update weights associated with early inputs.
Why This Matters for Long Sequences
Imagine processing the sentence: "The cat, which was sitting on the comfortable mat near the window overlooking the garden, was happy." To predict "was," the RNN needs to remember "cat" from 15+ words ago. But with vanishing gradients, the network can't learn this long-range dependency—the error signal from "was" barely reaches back to "cat."
Mathematically, gradients decay exponentially with sequence length because BPTT computes products of Jacobian matrices across time steps. When dominant eigenvalues are less than 1 (common with activation functions like tanh or sigmoid), gradients shrink toward zero.
This limitation motivated the development of specialized architectures (which you'll learn later) that can maintain gradient flow across longer sequences.
Key Takeaway: The vanishing gradient problem prevents standard RNNs from learning long-term dependencies because gradients decay exponentially as they propagate backward through many time steps, making it nearly impossible to train on relationships spanning more than ~10 time steps.