Initialization for Different Activation Functions
What you'll learn: How to match the right weight initialization scheme to your choice of activation function for optimal training.
Why Initialization Must Match Your Activation
Not all activation functions behave the same way mathematically, so they need different initialization strategies. Using the wrong initialization with a particular activation can lead to vanishing or exploding gradients, slowing or preventing learning entirely.
Think of it like tuning a musical instrument: different instruments (activation functions) need different tuning approaches (initialization schemes) to produce harmonious results.
The Key Matching Rules
ReLU and Its Variants (Leaky ReLU, PReLU)
Use He Initialization. ReLU zeros out negative values, effectively "killing" half the neurons' gradient flow. He initialization accounts for this by using a larger variance factor (√(2/n)) to compensate for the lost signal.
Sigmoid and Tanh
Use Xavier/Glorot Initialization. These functions are symmetric around zero and saturate on both ends. Xavier initialization preserves variance equally in forward and backward passes, using √(1/n) or √(2/(n_in + n_out)).
GELU and Swish/SiLU
Use He Initialization or slightly modified Xavier. These smooth, non-monotonic activations behave similarly to ReLU for positive values but allow small negative gradients. He initialization works well, though some practitioners use values between Xavier and He depending on the network depth.
Why This Matters
Mismatched initialization can cause:
- Vanishing gradients: Signals shrink to zero through deep layers
- Exploding gradients: Signals grow uncontrollably
- Slow convergence: Network takes much longer to learn
Key Takeaway: Always pair ReLU variants with He initialization, sigmoid/tanh with Xavier initialization, and modern activations like GELU/Swish with He or a hybrid approach—proper matching ensures stable gradient flow from the start.