Course contentsShow
← Machine Learning and Deep Learning
Lesson 654 of 3,53816. Activation Functions and Weight InitializationFree lesson

ReLU: The Rectified Linear Unit Revolution

Why ReLU became the default activation and how it solves vanishing gradient problems.

ReLU: The Rectified Linear Unit Revolution

What you'll learn: Why ReLU became the default activation function in deep learning and how it solves critical training problems.

The Problem ReLU Solved

Remember how sigmoid and tanh suffer from vanishing gradients? When their inputs get large (positive or negative), their gradients become extremely small—nearly zero. During backpropagation, these tiny gradients multiply together through layers, making deep networks almost impossible to train. Early layers receive practically no learning signal.

What is ReLU?

The Rectified Linear Unit (ReLU) is surprisingly simple:

f(x) = max(0, x)

If the input is positive, output it unchanged. If negative, output zero. That's it—no exponentials, no complex curves.

Why ReLU Revolutionized Deep Learning

1. No Vanishing Gradient (for positive values): When x > 0, the gradient is exactly 1. No shrinkage, no saturation. Gradients flow cleanly backward through many layers.

2. Computational Efficiency: ReLU is just a comparison and threshold—far faster than computing exponentials in sigmoid or tanh.

3. Sparse Activation: About half the neurons output zero for any input, creating sparse representations that can be more efficient and interpretable.

The Analogy

Think of gradient flow like water through pipes. Sigmoid/tanh are narrow, twisting pipes where water pressure drops dramatically. ReLU is a straight, wide pipe (when positive)—water flows at full pressure. This lets signals reach deep into your network.

The Trade-off

ReLU isn't perfect—neurons can "die" when they always output zero (gradient is also zero there), but this is manageable and vastly outweighed by its benefits for training deep networks.

Key Takeaway: ReLU's simple max(0, x) formula solved the vanishing gradient crisis by providing constant gradients for positive inputs, enabling the training of truly deep neural networks and becoming the default activation function.