Batch Normalization: Core Concept
What you'll learn: How Batch Normalization normalizes activations across the batch dimension to stabilize training and enable higher learning rates.
The Problem BatchNorm Solves
As gradients flow backward through deep networks, activations in intermediate layers can shift dramatically during training. This phenomenon, called internal covariate shift, forces each layer to constantly adapt to changing input distributions. It's like trying to learn while the ground keeps moving beneath your feet—progress becomes slow and unstable.
How Batch Normalization Works
Batch Normalization (BatchNorm) normalizes the activations of each layer across the mini-batch dimension. For each feature in a layer, BatchNorm:
- Computes the mean and variance across all examples in the mini-batch
- Normalizes the activations to have zero mean and unit variance
- Scales and shifts the normalized values using learnable parameters (γ and β)
Think of it like standardizing test scores: first convert everyone's score to a z-score (mean=0, std=1), then allow the network to learn the optimal scale and offset for each feature.
Why It Helps
By keeping activations in a consistent range:
- Gradient flow improves: prevents vanishing/exploding gradients
- Higher learning rates become safe: the normalized distribution is more stable
- Reduces dependence on initialization: normalization compensates for poor weight initialization
- Acts as regularization: the batch statistics add noise during training (similar to dropout's effect)
Key Takeaway: Batch Normalization normalizes layer activations using mini-batch statistics (mean and variance), stabilizing training, allowing higher learning rates, and reducing sensitivity to weight initialization—making deep networks much easier to train.