Course contentsShow
← Machine Learning and Deep Learning
Lesson 1204 of 3,53827. Pretrained Language Models: GPT Family and BeyondFree lesson

Layer Normalization Placement in GPT Models

Understanding pre-normalization vs post-normalization and its impact on training stability at scale.

Layer Normalization Placement in GPT Models

You'll learn: How moving layer normalization before versus after attention/feedforward blocks dramatically affects training stability in large language models like GPT.

The Two Approaches

In transformer architectures, layer normalization can be placed in two positions:

  1. Post-normalization (Post-LN): Normalize after the residual connection — the original Transformer design
  2. Pre-normalization (Pre-LN): Normalize before the attention or feedforward block — GPT-2 and modern practice

Why This Matters at Scale

Think of layer normalization like a thermostat controlling heat flow through your model. In the original Transformer, normalization happened after adding the residual (Post-LN). This works for smaller models, but as you scale to billions of parameters:

Post-LN problems:

  • Gradients can explode early in training because unnormalized activations pass through multiple layers
  • Requires very careful learning rate warmup
  • Training becomes unstable at large scale

Pre-LN advantages:

  • Normalizes inputs to each block, stabilizing what flows through attention and feedforward layers
  • Residual connection bypasses normalization, creating a cleaner gradient highway
  • Much more stable training, especially for deep networks
  • GPT-2 and GPT-3 use Pre-LN exclusively

The Practical Impact

GPT-2's switch to Pre-LN was crucial for scaling from GPT-1's 117M parameters to 1.5B. It allowed training without aggressive warmup schedules and made loss curves smoother. You're essentially preventing activations from growing too large before they pass through complex computations.

Key Takeaway: Pre-normalization (normalizing before attention/feedforward blocks) provides critical training stability for large-scale GPT models, enabling the deep architectures needed for modern language understanding without gradient explosions.