Course contentsShow
← Machine Learning and Deep Learning
Lesson 762 of 3,53818. Regularization Techniques for Deep NetworksFree lesson

Normalization Layer Placement and Architecture

Determine optimal placement of normalization layers: before or after activation, and how this affects residual connections.

Normalization Layer Placement and Architecture

What You'll Learn

You'll discover where to place normalization layers in your network architecture and how this decision impacts learning dynamics, especially with residual connections.

The Placement Puzzle

Now that you understand various normalization techniques (Batch Norm, Layer Norm, etc.), the critical question becomes: where exactly do you put them? This seemingly small decision profoundly affects gradient flow, training stability, and final performance.

Before vs After Activation

You have two main options:

Pre-activation (Norm → Activation → Weights):

Input → Normalization → Activation → Linear/Conv → Output

Post-activation (Weights → Activation → Norm):

Input → Linear/Conv → Activation → Normalization → Output

The pre-activation order is generally preferred in modern architectures. Why? Normalization directly affects the weights' inputs, stabilizing gradients before they encounter nonlinearities. The activation function then operates on normalized, well-behaved data.

Post-activation was common in early Batch Norm papers but can be problematic: activations like ReLU may kill neurons before normalization has a chance to help.

Impact on Residual Connections

In residual networks (which use skip connections for gradient flow), placement matters even more:

Post-activation residual block:

x → [Conv → BN → ReLU] → add x → output

Pre-activation residual block (preferred):

x → [BN → ReLU → Conv] → add x → output

Pre-activation keeps the identity path clean—the skip connection passes raw gradients unchanged, while normalization and activation happen inside the residual branch. This dramatically improves gradient flow through very deep networks.

Key Takeaway: Place normalization layers before activations in modern architectures (pre-activation order), especially in residual networks, to maintain clean gradient highways and stable training dynamics.