Course contentsShow
← Machine Learning and Deep Learning
Lesson 1617 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Parameter Initialization for Stability

Learn initialization strategies that prevent activation variance explosion in very deep transformer stacks.

Parameter Initialization for Stability

What you'll learn: How to initialize weights in very deep transformers so layer outputs maintain consistent variance, preventing training collapse or exploding activations.

Why Initialization Matters in Deep Transformers

When you stack dozens or hundreds of transformer layers (as in GPT-3 or larger models), naive weight initialization causes a snowball effect. If each layer slightly amplifies its input variance, by layer 50 the activations can explode into infinity. If each layer slightly shrinks variance, deep layers receive near-zero signals and learn nothing.

Think of it like a game of telephone through 100 people: if each person speaks 10% louder, the final message is deafening. If each whispers 10% softer, the message vanishes.

Initialization Strategies for Stability

Standard approaches adapted for transformers:

  • Scaled initialization: Initialize weights with variance proportional to 1/fan_in (Xavier) or 2/fan_in (Kaiming/He), ensuring each layer's output variance roughly matches its input variance
  • Residual path scaling: Since transformers use residual connections (x + attention(x) + ffn(x)), initialize attention and FFN outputs with smaller variance (often scaled by 1/sqrt(num_layers)) so residuals don't dominate
  • Layer-specific scaling: Initialize parameters in deeper layers with progressively smaller values to account for accumulated depth

Modern practice: Many LLMs use combinations: normal initialization for embeddings, scaled-down initialization for attention projection matrices and FFN outputs, and careful tuning of pre-norm layer normalization parameters.

The goal is variance preservation: keeping activation magnitudes stable throughout the forward pass so gradients flow cleanly backward during training.

Key Takeaway: Proper parameter initialization in deep transformers prevents activation variance from exploding or vanishing through the stack, enabling stable training of models with hundreds of layers by ensuring each layer preserves signal magnitude through carefully scaled weight values.