The Exploding Gradient Problem
What You'll Learn
You'll understand how gradients can blow up exponentially during backpropagation, making neural network training impossible.
The Core Problem
While the vanishing gradient problem causes gradients to shrink toward zero, the exploding gradient problem is the opposite nightmare: gradients grow exponentially larger as they backpropagate through layers.
During backpropagation, the chain rule multiplies gradients from each layer. When these multiplied values are consistently greater than 1, they compound exponentially as they move backward through deep networks. This causes gradient values to explode into astronomically large numbers.
Why This Breaks Training
When gradients explode:
- Weight updates become massive — instead of small adjustments, your network makes wild, erratic jumps
- NaN (Not a Number) values appear — gradients become so large they exceed numerical limits
- Loss diverges — instead of decreasing, your loss shoots to infinity
- Training becomes unstable — the network oscillates wildly and never converges
The Multiplication Chain
Think of it like compound interest gone wrong. If you multiply 1.5 × 1.5 × 1.5... across 20 layers, you get a number over 3,000. Now imagine those are gradient multipliers — your gradients explode from reasonable values to unmanageable ones.
This is especially problematic with:
- Very deep networks (many layers to multiply through)
- Poor weight initialization (weights too large)
- Certain activation functions (those with derivatives > 1 in ranges)
- High learning rates (amplifying already-large updates)
Key Takeaway: The exploding gradient problem occurs when repeated gradient multiplication during backpropagation causes values to grow exponentially, leading to NaN values and training collapse. Proper initialization, gradient clipping, and architectural choices help prevent this.