Course contentsShow
← Machine Learning and Deep Learning
Lesson 672 of 3,53816. Activation Functions and Weight InitializationFree lesson

Layer-Specific Initialization

Special initialization for batch norm, layer norm, and residual connections to stabilize training.

Layer-Specific Initialization

You'll learn: How to initialize special layer types (batch norm, layer norm, residual connections) differently from standard layers to maintain training stability.

Why Not All Layers Are Created Equal

So far, we've covered initialization strategies like Xavier and He that work well for standard fully connected layers with various activation functions. But modern deep networks include specialized components—batch normalization, layer normalization, and residual connections—that process information differently and need their own initialization rules.

Think of it like preparing different types of engines: a car engine and a jet engine both need fuel, but they require different startup procedures to run smoothly.

Layer-Specific Rules

Batch Norm and Layer Norm

These normalization layers include learnable scale (γ) and shift (β) parameters. The common practice:

  • Initialize γ (gamma) to 1: preserves variance through the layer initially
  • Initialize β (beta) to 0: starts with zero shift, letting normalization work cleanly

This ensures the layer initially acts as a simple pass-through with normalization, then learns adjustments during training.

Residual Connections

In networks with skip connections (like ResNets), the layers added to the residual path are often initialized with smaller weights or even zeros. This makes the network behave like a shallow network initially, gradually learning to use deeper paths. Some implementations scale the final layer of each residual block by a small constant or initialize it to zero.

The Stabilization Goal

These special initializations prevent early training instabilities that occur when these layers start with arbitrary values, ensuring gradients flow smoothly from the beginning.

Key Takeaway: Specialized layers (batch norm, layer norm, residual blocks) need tailored initialization—typically identity-like behavior (scale=1, shift=0, or near-zero weights)—to ensure stable gradient flow and effective training from step one.