Course contentsShow
← Machine Learning and Deep Learning
Lesson 670 of 3,53816. Activation Functions and Weight InitializationFree lesson

Initialization for Different Activation Functions

Matching initialization schemes to activation functions: ReLU variants, GELU, and Swish.

Initialization for Different Activation Functions

What you'll learn: How to match the right weight initialization scheme to your choice of activation function for optimal training.

Why Initialization Must Match Your Activation

Not all activation functions behave the same way mathematically, so they need different initialization strategies. Using the wrong initialization with a particular activation can lead to vanishing or exploding gradients, slowing or preventing learning entirely.

Think of it like tuning a musical instrument: different instruments (activation functions) need different tuning approaches (initialization schemes) to produce harmonious results.

The Key Matching Rules

ReLU and Its Variants (Leaky ReLU, PReLU)

Use He Initialization. ReLU zeros out negative values, effectively "killing" half the neurons' gradient flow. He initialization accounts for this by using a larger variance factor (√(2/n)) to compensate for the lost signal.

Sigmoid and Tanh

Use Xavier/Glorot Initialization. These functions are symmetric around zero and saturate on both ends. Xavier initialization preserves variance equally in forward and backward passes, using √(1/n) or √(2/(n_in + n_out)).

GELU and Swish/SiLU

Use He Initialization or slightly modified Xavier. These smooth, non-monotonic activations behave similarly to ReLU for positive values but allow small negative gradients. He initialization works well, though some practitioners use values between Xavier and He depending on the network depth.

Why This Matters

Mismatched initialization can cause:

  • Vanishing gradients: Signals shrink to zero through deep layers
  • Exploding gradients: Signals grow uncontrollably
  • Slow convergence: Network takes much longer to learn

Key Takeaway: Always pair ReLU variants with He initialization, sigmoid/tanh with Xavier initialization, and modern activations like GELU/Swish with He or a hybrid approach—proper matching ensures stable gradient flow from the start.