The Vanishing Gradient Problem in RNNs
What you'll learn: Why standard RNNs struggle to remember information from many time steps ago, making them unable to learn long-term dependencies.
The Core Problem
When training RNNs with backpropagation through time (BPTT), gradients must flow backward through many time steps. At each step, the gradient gets multiplied by the weight matrix and passes through activation function derivatives. If these values are small (typically less than 1), repeated multiplication causes the gradient to shrink exponentially.
Think of it like making photocopies of photocopies: each generation loses quality. After 20+ copies, the image becomes unrecognizable. Similarly, after flowing backward through many time steps, gradients become vanishingly small—often approaching zero.
Why This Matters
When gradients vanish:
- Early time steps receive almost no learning signal – the network can't adjust weights that affect distant past inputs
- Long-term dependencies cannot be learned – if understanding a sentence requires remembering a word from 50 positions back, standard RNNs fail
- Training stalls – weight updates become negligibly small, halting progress
The Mathematical Culprit
During BPTT, the gradient at time step t depends on a chain of derivatives multiplied together. If the recurrent weight matrix has values less than 1 and you're using activation functions like tanh or sigmoid (which have derivatives ≤ 1), multiplying many small numbers together yields exponentially smaller results.
For example: 0.5 × 0.5 × 0.5 × ... (repeated 30 times) ≈ 0.0000000009
This explains why vanilla RNNs work well for short sequences but fail on tasks requiring memory of 20+ steps back.
Key Takeaway: The vanishing gradient problem occurs when gradients shrink exponentially during backpropagation through many time steps, preventing standard RNNs from learning long-term dependencies—this fundamental limitation motivated the development of specialized architectures.