Layer Normalization vs Batch Normalization
What you'll learn: When to choose Layer Normalization over Batch Normalization (and vice versa) based on your network architecture and batch size constraints.
The Core Difference
Both techniques normalize activations, but across different dimensions:
- Batch Normalization normalizes across the batch dimension (all examples for each feature)
- Layer Normalization normalizes across the feature dimension (all features for each example)
This fundamental difference determines when each works best.
When to Use Layer Normalization
RNNs and Transformers: These process sequences where each timestep has different statistics. BatchNorm struggles here because it assumes similar distributions across the batch, but sequence elements vary wildly. LayerNorm normalizes each timestep independently, making it ideal for sequential models.
Small Batches: When your batch size is 1 or very small (due to memory constraints or large models), BatchNorm's statistics become unreliable—you can't accurately estimate mean/variance from 2-3 examples. LayerNorm doesn't depend on batch size at all.
Online/Real-time Inference: LayerNorm computes statistics from the current example alone, avoiding the train/inference mode complexity of BatchNorm's running averages.
When to Use Batch Normalization
CNNs and Vision Tasks: BatchNorm excels in convolutional networks where spatial features should have consistent statistics across examples (e.g., edge detectors). It leverages batch statistics to stabilize training.
Large Batches: With batch sizes of 32+ examples, BatchNorm's mean/variance estimates are reliable and provide strong regularization effects through batch-dependent noise.
Standard Supervised Learning: When you have i.i.d. data and sufficient batch size, BatchNorm's proven track record in vision tasks makes it the default choice.
Analogy
Think of BatchNorm as grading on a curve (comparing students in a class), while LayerNorm is grading against a rubric (evaluating each student's answers independently).
Key Takeaway: Use LayerNorm for RNNs, Transformers, and small-batch scenarios; use BatchNorm for CNNs with large batches and vision tasks where batch statistics are meaningful and stable.