Course contentsShow
← Machine Learning and Deep Learning
Lesson 675 of 3,53816. Activation Functions and Weight InitializationFree lesson

The Vanishing Gradient Problem

Understanding why gradients can exponentially decay in deep networks, making early layers difficult to train.

The Vanishing Gradient Problem

What you'll learn: Why deep neural networks struggle to train their early layers due to exponentially shrinking gradients.

The Core Issue

When training deep neural networks with backpropagation, gradients must flow backward from the output layer through all hidden layers to update weights. In very deep networks, these gradients can become exponentially smaller as they propagate backward through each layer. By the time gradients reach the early layers, they're so tiny (approaching zero) that weight updates become negligible—essentially stopping those layers from learning.

Why This Happens

Remember that backpropagation uses the chain rule, multiplying gradients layer by layer. If each layer's gradient is less than 1 (common with activation functions like sigmoid or tanh), repeatedly multiplying many small numbers causes the product to shrink exponentially:

0.3 × 0.25 × 0.4 × 0.2 × ... → practically zero

This is especially problematic with sigmoid and tanh activations, whose derivatives are always less than 1 (sigmoid's maximum derivative is 0.25).

An Analogy

Think of passing a message through a chain of whispering people. If each person only passes along 25% of what they heard, by the time the message reaches the 10th person, barely anything remains. Early layers become "deaf" to the learning signal from the loss function.

Consequences

  • Early layers train extremely slowly or not at all
  • Network struggles to learn complex hierarchical features
  • Deep networks perform worse than shallow ones (before modern solutions)

Key Takeaway: The vanishing gradient problem occurs when gradients exponentially decay as they backpropagate through deep networks, preventing early layers from learning—a fundamental challenge that motivated modern activation functions (like ReLU) and initialization strategies.