Leaky ReLU: Fixing Dead Neurons
You'll learn how Leaky ReLU solves the dying ReLU problem by allowing small negative slopes instead of zero outputs.
The Problem Recap
Remember the dying ReLU problem? When inputs are negative, standard ReLU outputs exactly zero, and its gradient is also zero. This means neurons can get permanently "stuck" — they stop learning entirely because no gradient flows backward through them during training.
The Leaky ReLU Solution
Leaky ReLU makes a simple but powerful modification: instead of outputting zero for negative inputs, it outputs a small negative value proportional to the input.
The mathematical definition:
f(x) = max(αx, x)
Where α is a small constant, typically 0.01. This means:
- For positive inputs: output = input (just like ReLU)
- For negative inputs: output = 0.01 × input (a small negative slope)
Why This Works
Think of it like a safety valve. Even when a neuron receives negative inputs, it still produces a small non-zero output. More importantly, the gradient is never exactly zero — it's either 1 (for positive inputs) or α (for negative inputs).
This tiny negative slope keeps gradient flowing backward during backpropagation, so the neuron can adjust its weights and potentially "wake up" again. The neuron never truly dies.
Practical Impact
With Leaky ReLU, your network becomes more robust. Neurons that would have died with standard ReLU now maintain the ability to learn and recover, leading to more stable training and often better final performance.
Key Takeaway: Leaky ReLU prevents dying neurons by using a small negative slope (typically 0.01) instead of zero for negative inputs, ensuring gradients always flow and neurons can always learn.