Parameter Initialization for Stability
What you'll learn: How to initialize weights in very deep transformers so layer outputs maintain consistent variance, preventing training collapse or exploding activations.
Why Initialization Matters in Deep Transformers
When you stack dozens or hundreds of transformer layers (as in GPT-3 or larger models), naive weight initialization causes a snowball effect. If each layer slightly amplifies its input variance, by layer 50 the activations can explode into infinity. If each layer slightly shrinks variance, deep layers receive near-zero signals and learn nothing.
Think of it like a game of telephone through 100 people: if each person speaks 10% louder, the final message is deafening. If each whispers 10% softer, the message vanishes.
Initialization Strategies for Stability
Standard approaches adapted for transformers:
- Scaled initialization: Initialize weights with variance proportional to
1/fan_in(Xavier) or2/fan_in(Kaiming/He), ensuring each layer's output variance roughly matches its input variance - Residual path scaling: Since transformers use residual connections (
x + attention(x) + ffn(x)), initialize attention and FFN outputs with smaller variance (often scaled by1/sqrt(num_layers)) so residuals don't dominate - Layer-specific scaling: Initialize parameters in deeper layers with progressively smaller values to account for accumulated depth
Modern practice: Many LLMs use combinations: normal initialization for embeddings, scaled-down initialization for attention projection matrices and FFN outputs, and careful tuning of pre-norm layer normalization parameters.
The goal is variance preservation: keeping activation magnitudes stable throughout the forward pass so gradients flow cleanly backward during training.
Key Takeaway: Proper parameter initialization in deep transformers prevents activation variance from exploding or vanishing through the stack, enabling stable training of models with hundreds of layers by ensuring each layer preserves signal magnitude through carefully scaled weight values.