Layer-Specific Initialization
You'll learn: How to initialize special layer types (batch norm, layer norm, residual connections) differently from standard layers to maintain training stability.
Why Not All Layers Are Created Equal
So far, we've covered initialization strategies like Xavier and He that work well for standard fully connected layers with various activation functions. But modern deep networks include specialized components—batch normalization, layer normalization, and residual connections—that process information differently and need their own initialization rules.
Think of it like preparing different types of engines: a car engine and a jet engine both need fuel, but they require different startup procedures to run smoothly.
Layer-Specific Rules
Batch Norm and Layer Norm
These normalization layers include learnable scale (γ) and shift (β) parameters. The common practice:
- Initialize γ (gamma) to 1: preserves variance through the layer initially
- Initialize β (beta) to 0: starts with zero shift, letting normalization work cleanly
This ensures the layer initially acts as a simple pass-through with normalization, then learns adjustments during training.
Residual Connections
In networks with skip connections (like ResNets), the layers added to the residual path are often initialized with smaller weights or even zeros. This makes the network behave like a shallow network initially, gradually learning to use deeper paths. Some implementations scale the final layer of each residual block by a small constant or initialize it to zero.
The Stabilization Goal
These special initializations prevent early training instabilities that occur when these layers start with arbitrary values, ensuring gradients flow smoothly from the beginning.
Key Takeaway: Specialized layers (batch norm, layer norm, residual blocks) need tailored initialization—typically identity-like behavior (scale=1, shift=0, or near-zero weights)—to ensure stable gradient flow and effective training from step one.