Course contentsShow
← Machine Learning and Deep Learning
Lesson 876 of 3,53820. Convolutional Neural Networks: FundamentalsFree lesson

Activation Functions in CNN Architectures

Examine ReLU and its variants as the standard activation for CNNs, understanding placement after convolutions and before pooling.

Activation Functions in CNN Architectures

What you'll learn: Where and why ReLU and its variants appear in CNN layers, and how they enable networks to learn complex patterns.

Why Activation Functions Matter in CNNs

Without activation functions, stacking convolutional layers would just create one big linear transformation—no matter how deep your network, it could only learn simple, straight-line relationships. Activation functions introduce non-linearity, allowing CNNs to learn curves, edges, textures, and complex visual patterns.

ReLU: The Standard Choice

ReLU (Rectified Linear Unit) is the dominant activation in modern CNNs. It's simple: f(x) = max(0, x). Any negative value becomes zero; positive values pass through unchanged.

Think of ReLU like a filter that blocks negative signals but amplifies positive ones—similar to how our neurons fire only when stimulation exceeds a threshold.

Where to Place Activations

The standard CNN pattern is:

  1. Convolution (extracts features)
  2. Activation (ReLU introduces non-linearity)
  3. Pooling (downsamples spatially)

You apply the activation immediately after each convolution, before pooling. This ensures each convolutional layer can learn non-linear patterns before reducing spatial dimensions.

ReLU Variants You'll Encounter

  • Leaky ReLU: Allows small negative values (f(x) = max(0.01x, x)) to prevent "dead neurons"
  • ELU: Smooth negative values for better gradient flow
  • Parametric ReLU: Learns the negative slope during training

These variants address ReLU's weakness: neurons can "die" (permanently output zero) if they consistently produce negative values.

Why ReLU Won

ReLU is computationally cheap, doesn't saturate for positive values (unlike sigmoid/tanh), and empirically works better for deep networks—gradients flow more easily during backpropagation.

Key Takeaway: ReLU and its variants are placed after every convolution in CNNs to introduce non-linearity, enabling the network to learn complex visual features rather than just linear combinations of pixels.