Course contentsShow
← Machine Learning and Deep Learning
Lesson 725 of 3,53817. Optimization for Deep LearningFree lesson

The Exploding Gradient Problem

Understanding why gradients can grow exponentially during backpropagation and destabilize training.

The Exploding Gradient Problem

What you'll learn: How gradients can become dangerously large during backpropagation and wreak havoc on your training process.

The Core Issue

During backpropagation, gradients flow backward through your network layer by layer. At each step, the chain rule multiplies gradients together. When many of these multiplied values are greater than 1, the gradient grows exponentially as it travels deeper into the network—like compound interest on steroids.

Imagine whispering a message through a chain of people, but each person shouts twice as loud as the person before them. By the time your message reaches the 10th person, it's deafening. That's exploding gradients.

Why It Happens

The exploding gradient problem occurs when:

  • Weight values are too large: Multiplying big numbers repeatedly makes them huge
  • Deep networks amplify the effect: More layers = more multiplications
  • Poor initialization: Starting with inappropriate weight scales
  • Certain activation functions: Some can contribute to gradient multiplication

The Consequences

When gradients explode:

  • Weight updates become massive and erratic
  • Loss values oscillate wildly or shoot to infinity/NaN
  • The model fails to converge—it literally can't learn
  • Training becomes numerically unstable

Detecting Exploding Gradients

Watch for these warning signs:

  • Loss suddenly jumping to very large values
  • Weight values growing uncontrollably
  • NaN (Not a Number) appearing in your outputs
  • Model performance deteriorating dramatically during training

Key Takeaway: The exploding gradient problem occurs when repeated gradient multiplications during backpropagation cause gradients to grow exponentially, leading to unstable training and preventing your model from learning effectively.