Group Normalization
You'll learn: How Group Normalization bridges the gap between Batch Norm and Layer Norm by normalizing within channel groups, making it stable even with tiny batches.
The Core Idea
You've seen Batch Normalization (normalizes across the batch dimension) and Layer Normalization (normalizes across all channels for each sample independently). Group Normalization (GroupNorm) takes a middle-ground approach: it divides the channels into groups and normalizes within each group independently for each sample.
Think of it like organizing a classroom. Batch Norm grades students by comparing everyone in the class. Layer Norm grades each student by comparing all their subjects together. Group Norm divides subjects into groups (sciences, humanities, arts) and grades within each group separately.
Why Group Normalization?
Batch Norm struggles when batch sizes are small (like 1-2 samples) because statistics computed from tiny batches are noisy and unreliable. Layer Norm treats all channels equally, which isn't always ideal for convolutional networks where channels represent different features.
Group Normalization solves this by:
- Not depending on batch size — statistics are computed per sample, per group
- Preserving some channel structure — related channels in a group share normalization statistics
- Working effectively with small batches — common in tasks like object detection or video processing where memory limits batch size
How It Works
For a layer with C channels, you divide them into G groups (each containing C/G channels). For each sample and each group, you compute mean and variance across spatial dimensions and the channels within that group, then normalize.
Key Takeaway: Group Normalization divides channels into groups and normalizes within each group independently per sample, offering Batch Norm's benefits without requiring large batches—ideal when batch size is constrained.