Course contentsShow
← Machine Learning and Deep Learning
Lesson 1005 of 3,53823. Recurrent Neural NetworksFree lesson

The Exploding Gradient Problem

How repeated multiplication can cause gradients to grow unboundedly and destabilize training.

The Exploding Gradient Problem

What you'll learn: How repeated multiplication during backpropagation can cause gradients to grow uncontrollably large, destabilizing RNN training.

The Core Problem

Remember how vanishing gradients happen when numbers smaller than 1 multiply repeatedly? The exploding gradient problem is the opposite scenario. During Backpropagation Through Time (BPTT), if the weights or gradients being multiplied are consistently larger than 1, they grow exponentially as they flow backward through time steps.

Think of it like compound interest gone wild: if you multiply 2 × 2 × 2 × 2 across just 10 time steps, you get 1,024. Across 50 steps? Over a quadrillion. In RNNs processing long sequences, this repeated multiplication causes gradients to become astronomically large.

Why It's Dangerous

Unlike vanishing gradients where learning simply slows down, exploding gradients actively destabilize training:

  • Weight updates become enormous (gradient × learning rate = huge number)
  • Loss suddenly spikes to NaN (Not a Number)
  • Model parameters overflow to infinity
  • Training becomes impossible to complete

The Telltale Signs

You'll know gradients are exploding when:

  • Loss suddenly jumps from reasonable values to infinity
  • You see NaN in your loss logs
  • Weights grow absurdly large during training
  • The model's predictions become completely nonsensical mid-training

The Solution Preview

While we'll learn specific solutions later, the most common fix is gradient clipping: capping gradient values at a maximum threshold before applying updates, preventing the runaway multiplication from destroying your model.

Key Takeaway: The exploding gradient problem occurs when repeated multiplication during BPTT causes gradients to grow unboundedly large, leading to training instability, NaN losses, and unusable models—the dangerous flip side of vanishing gradients.