Layer Normalization Fundamentals
What you'll learn: How Layer Normalization normalizes across features instead of batch samples, making it independent of batch size and ideal for sequential models.
The Batch Normalization Problem
Batch Normalization (which you just learned) normalizes across the batch dimension—it looks at the same feature across all samples in a mini-batch. This creates a dependency on batch size: small batches give noisy statistics, and at inference with a single sample, you must rely on stored running statistics.
Layer Normalization's Key Difference
Layer Normalization (LayerNorm) takes a completely different approach: it normalizes across all features within a single sample. Instead of asking "what's the mean of feature 5 across all batch samples?", it asks "what's the mean of all features for this one sample?"
Why This Matters
Think of each training example as a student taking a test. Batch Normalization compares all students' scores on question 1, then normalizes. Layer Normalization looks at one student's performance across all questions and normalizes their scores relative to their own average.
This makes LayerNorm:
- Batch-size independent: Works perfectly with batch size = 1
- Identical in training and inference: No need for running statistics
- Perfect for sequences: Each token in a sentence can be normalized independently
When to Use LayerNorm
LayerNorm shines in:
- Recurrent networks (RNNs, LSTMs) where batch sizes vary
- Transformers where sequence lengths differ
- Reinforcement learning with single-sample updates
The trade-off? LayerNorm doesn't benefit from batch-level statistics that sometimes help BatchNorm generalize better in computer vision tasks.
Key Takeaway: Layer Normalization normalizes across features within each sample independently, making it batch-size agnostic and the preferred choice for sequential models and variable-batch scenarios.