The Exploding Gradient Problem
What you'll learn: How repeated multiplication during backpropagation can cause gradients to grow uncontrollably large, destabilizing RNN training.
The Core Problem
Remember how vanishing gradients happen when numbers smaller than 1 multiply repeatedly? The exploding gradient problem is the opposite scenario. During Backpropagation Through Time (BPTT), if the weights or gradients being multiplied are consistently larger than 1, they grow exponentially as they flow backward through time steps.
Think of it like compound interest gone wild: if you multiply 2 × 2 × 2 × 2 across just 10 time steps, you get 1,024. Across 50 steps? Over a quadrillion. In RNNs processing long sequences, this repeated multiplication causes gradients to become astronomically large.
Why It's Dangerous
Unlike vanishing gradients where learning simply slows down, exploding gradients actively destabilize training:
- Weight updates become enormous (gradient × learning rate = huge number)
- Loss suddenly spikes to NaN (Not a Number)
- Model parameters overflow to infinity
- Training becomes impossible to complete
The Telltale Signs
You'll know gradients are exploding when:
- Loss suddenly jumps from reasonable values to infinity
- You see NaN in your loss logs
- Weights grow absurdly large during training
- The model's predictions become completely nonsensical mid-training
The Solution Preview
While we'll learn specific solutions later, the most common fix is gradient clipping: capping gradient values at a maximum threshold before applying updates, preventing the runaway multiplication from destroying your model.
Key Takeaway: The exploding gradient problem occurs when repeated multiplication during BPTT causes gradients to grow unboundedly large, leading to training instability, NaN losses, and unusable models—the dangerous flip side of vanishing gradients.