Course contentsShow
← Machine Learning and Deep Learning
Lesson 909 of 3,53821. CNN ArchitecturesFree lesson

Pre-Activation Residual Blocks

Placing batch normalization and ReLU before convolutions for improved gradient flow.

Pre-Activation Residual Blocks

What you'll learn: How reordering batch normalization and ReLU before convolutions in residual blocks improves gradient flow and training stability.

The Original Problem

In the standard residual block you've seen, the pattern goes: convolution → batch normalization → ReLU. The skip connection adds the identity directly to this processed output. While revolutionary, researchers discovered this arrangement could still be improved.

Pre-Activation: Flipping the Order

Pre-activation residual blocks restructure the operations so that batch normalization and ReLU happen before the convolution layers, not after. The pattern becomes: batch normalization → ReLU → convolution.

Think of it like preparing ingredients before cooking versus seasoning after. By "activating" the data first (normalizing and applying ReLU), you ensure the convolution receives cleaner, more normalized input.

Why This Matters

This reordering creates a critical advantage: the skip connection now bypasses all transformations completely. The identity path flows through without encountering any non-linearities or normalizations. This creates an unobstructed highway for gradients during backpropagation.

In the original ResNet, gradients still had to flow through batch norm and ReLU on the residual path. With pre-activation, gradients can flow backward through the pure identity connection without any modifications, making very deep networks (100+ layers) significantly easier to train.

The Structure

The pre-activation block looks like:

  • Input → [BN → ReLU → Conv → BN → ReLU → Conv] → Add with identity → Output

Notice the identity adds to the output after all activations have been applied to the residual path.

Key Takeaway: Pre-activation residual blocks place batch normalization and ReLU before convolutions, creating a completely clean identity path that enables superior gradient flow in very deep networks.