Layer Normalization Placement in GPT Models
You'll learn: How moving layer normalization before versus after attention/feedforward blocks dramatically affects training stability in large language models like GPT.
The Two Approaches
In transformer architectures, layer normalization can be placed in two positions:
- Post-normalization (Post-LN): Normalize after the residual connection — the original Transformer design
- Pre-normalization (Pre-LN): Normalize before the attention or feedforward block — GPT-2 and modern practice
Why This Matters at Scale
Think of layer normalization like a thermostat controlling heat flow through your model. In the original Transformer, normalization happened after adding the residual (Post-LN). This works for smaller models, but as you scale to billions of parameters:
Post-LN problems:
- Gradients can explode early in training because unnormalized activations pass through multiple layers
- Requires very careful learning rate warmup
- Training becomes unstable at large scale
Pre-LN advantages:
- Normalizes inputs to each block, stabilizing what flows through attention and feedforward layers
- Residual connection bypasses normalization, creating a cleaner gradient highway
- Much more stable training, especially for deep networks
- GPT-2 and GPT-3 use Pre-LN exclusively
The Practical Impact
GPT-2's switch to Pre-LN was crucial for scaling from GPT-1's 117M parameters to 1.5B. It allowed training without aggressive warmup schedules and made loss curves smoother. You're essentially preventing activations from growing too large before they pass through complex computations.
Key Takeaway: Pre-normalization (normalizing before attention/feedforward blocks) provides critical training stability for large-scale GPT models, enabling the deep architectures needed for modern language understanding without gradient explosions.