Course contentsShow
← Machine Learning and Deep Learning
Lesson 656 of 3,53816. Activation Functions and Weight InitializationFree lesson

Leaky ReLU: Fixing Dead Neurons

How allowing small negative slopes prevents neurons from dying during training.

Leaky ReLU: Fixing Dead Neurons

You'll learn how Leaky ReLU solves the dying ReLU problem by allowing small negative slopes instead of zero outputs.

The Problem Recap

Remember the dying ReLU problem? When inputs are negative, standard ReLU outputs exactly zero, and its gradient is also zero. This means neurons can get permanently "stuck" — they stop learning entirely because no gradient flows backward through them during training.

The Leaky ReLU Solution

Leaky ReLU makes a simple but powerful modification: instead of outputting zero for negative inputs, it outputs a small negative value proportional to the input.

The mathematical definition:

f(x) = max(αx, x)

Where α is a small constant, typically 0.01. This means:

  • For positive inputs: output = input (just like ReLU)
  • For negative inputs: output = 0.01 × input (a small negative slope)

Why This Works

Think of it like a safety valve. Even when a neuron receives negative inputs, it still produces a small non-zero output. More importantly, the gradient is never exactly zero — it's either 1 (for positive inputs) or α (for negative inputs).

This tiny negative slope keeps gradient flowing backward during backpropagation, so the neuron can adjust its weights and potentially "wake up" again. The neuron never truly dies.

Practical Impact

With Leaky ReLU, your network becomes more robust. Neurons that would have died with standard ReLU now maintain the ability to learn and recover, leading to more stable training and often better final performance.

Key Takeaway: Leaky ReLU prevents dying neurons by using a small negative slope (typically 0.01) instead of zero for negative inputs, ensuring gradients always flow and neurons can always learn.