Course contentsShow
← Machine Learning and Deep Learning
Lesson 655 of 3,53816. Activation Functions and Weight InitializationFree lesson

The Dying ReLU Problem

Understanding when ReLU neurons stop learning and why negative inputs cause issues.

The Dying ReLU Problem

What you'll learn: Why ReLU neurons can permanently stop learning during training and how negative inputs cause this critical issue.

The Problem with Zero Gradients

Remember that ReLU outputs zero for all negative inputs: f(x) = max(0, x). This seems harmless until you consider what happens during backpropagation.

When a ReLU neuron receives negative input, it outputs zero. More importantly, its gradient is also zero for that input. During backpropagation, this zero gradient means no weight updates occur for that neuron.

When Neurons Die

A "dead" neuron is one that consistently outputs zero across all training examples. Here's how it happens:

  1. A neuron's weights shift such that its inputs become negative for most/all data points
  2. ReLU outputs zero every time
  3. Gradients flowing back are always zero
  4. Weights never update (they're "stuck")
  5. The neuron contributes nothing to learning — it's effectively dead

Common causes:

  • Large learning rates: Weights jump too far during updates, landing in the negative region
  • Poor initialization: Starting weights produce mostly negative pre-activations
  • Data characteristics: If input features are predominantly negative, neurons are more vulnerable

Why This Matters

Dead neurons waste model capacity. In severe cases, large portions of your network can die, dramatically reducing its expressive power. You might train a 1000-neuron network but effectively use only 300 active neurons.

Key Takeaway: The Dying ReLU problem occurs when neurons get stuck outputting zero because their inputs remain negative, causing zero gradients that prevent any further learning — essentially creating permanent dead zones in your network.