Pre-normalization vs Post-normalization
What you'll learn: How the placement of LayerNorm in transformer blocks dramatically affects training stability in deep language models.
The Core Difference
In transformer architectures, LayerNorm can be placed in two positions relative to residual connections:
- Post-normalization (Post-norm): Original transformer design. LayerNorm comes after the residual addition:
LayerNorm(x + Sublayer(x)) - Pre-normalization (Pre-norm): Modern approach. LayerNorm comes before each sublayer:
x + Sublayer(LayerNorm(x))
Think of it like quality control in manufacturing: post-norm inspects the final assembled product, while pre-norm inspects components before assembly.
Why Pre-norm Stabilizes Deep Stacks
In decoder-only models like GPT (which stack dozens of layers), pre-norm has become the standard because:
-
Gradient flow: In post-norm, gradients must pass through the normalization layer during backpropagation, which can amplify or distort them. Pre-norm allows gradients to flow more directly through residual connections.
-
Signal magnitude: Post-norm can create large activation spikes when adding unnormalized sublayer outputs. Pre-norm keeps sublayer inputs bounded, preventing explosive growth.
-
Initialization sensitivity: Post-norm architectures require careful weight initialization and warmup strategies. Pre-norm is more forgiving, allowing higher learning rates from the start.
In practice, GPT-style models with 50+ layers nearly always use pre-norm because post-norm would require extensive hyperparameter tuning to avoid training collapse.
The Trade-off
Post-norm can achieve slightly better final performance when properly tuned in shallow networks, but the stability advantage of pre-norm makes it the clear winner for modern large language models with deep decoder stacks.
Key Takeaway: Pre-normalization places LayerNorm before each sublayer (not after residual addition), enabling stable training in deep decoder stacks by protecting gradient flow and preventing activation explosions.