Normalization Layer Placement and Architecture
What You'll Learn
You'll discover where to place normalization layers in your network architecture and how this decision impacts learning dynamics, especially with residual connections.
The Placement Puzzle
Now that you understand various normalization techniques (Batch Norm, Layer Norm, etc.), the critical question becomes: where exactly do you put them? This seemingly small decision profoundly affects gradient flow, training stability, and final performance.
Before vs After Activation
You have two main options:
Pre-activation (Norm → Activation → Weights):
Input → Normalization → Activation → Linear/Conv → Output
Post-activation (Weights → Activation → Norm):
Input → Linear/Conv → Activation → Normalization → Output
The pre-activation order is generally preferred in modern architectures. Why? Normalization directly affects the weights' inputs, stabilizing gradients before they encounter nonlinearities. The activation function then operates on normalized, well-behaved data.
Post-activation was common in early Batch Norm papers but can be problematic: activations like ReLU may kill neurons before normalization has a chance to help.
Impact on Residual Connections
In residual networks (which use skip connections for gradient flow), placement matters even more:
Post-activation residual block:
x → [Conv → BN → ReLU] → add x → output
Pre-activation residual block (preferred):
x → [BN → ReLU → Conv] → add x → output
Pre-activation keeps the identity path clean—the skip connection passes raw gradients unchanged, while normalization and activation happen inside the residual branch. This dramatically improves gradient flow through very deep networks.
Key Takeaway: Place normalization layers before activations in modern architectures (pre-activation order), especially in residual networks, to maintain clean gradient highways and stable training dynamics.