Course contentsShow
← Machine Learning and Deep Learning
Lesson 873 of 3,53820. Convolutional Neural Networks: FundamentalsFree lesson

Batch Normalization in CNNs

Understand how batch normalization normalizes activations across the batch dimension to stabilize training and enable higher learning rates.

Batch Normalization in CNNs

What You'll Learn

You'll learn how batch normalization normalizes activations across the batch dimension to stabilize training and enable higher learning rates in convolutional neural networks.

The Core Idea

After convolution and pooling operations, the distribution of activations can shift dramatically between layers and across training iterations. This phenomenon, called internal covariate shift, forces you to use small learning rates and careful weight initialization.

Batch normalization solves this by normalizing activations within each mini-batch. For each feature map (channel), it:

  1. Computes the mean and variance across the batch dimension
  2. Normalizes values to have mean=0 and variance=1
  3. Applies learnable scale (γ) and shift (β) parameters

Think of it like adjusting exam scores: if one class averages 50% and another 90%, you normalize both to a standard scale before comparing—but you still allow each to have its own "curve" via the learned parameters.

Why It Matters for CNNs

In CNNs, batch normalization is typically applied after convolution but before activation (though after activation also works). It operates independently on each channel, computing statistics across the batch and spatial dimensions.

Benefits:

  • Faster training: Allows 10-100× higher learning rates safely
  • Reduced sensitivity: Less dependence on careful weight initialization
  • Regularization effect: The noise from batch statistics acts like a mild regularizer
  • Gradient flow: Prevents vanishing/exploding gradients in deeper networks

The normalization happens separately for each channel, preserving the spatial structure that CNNs rely on while stabilizing the magnitude of activations flowing through the network.

Key Takeaway:
Batch normalization normalizes activations per channel across the batch, stabilizing training dynamics and enabling faster convergence with higher learning rates in CNNs.