He Initialization
What you'll learn: You'll discover why ReLU-based networks need a different weight initialization strategy than sigmoid or tanh networks, and how He initialization solves this problem.
Why Xavier Isn't Enough for ReLU
Remember Xavier/Glorot initialization? It assumed activations were zero-centered and approximately linear around zero (like tanh). But ReLU cuts off all negative values, setting them to zero. This changes the variance dynamics dramatically!
When ReLU zeros out roughly half the neurons, the variance of activations gets cut in half at each layer. If you use Xavier initialization with ReLU, your signal shrinks too quickly as it flows forward through the network.
The He Solution
He initialization (named after researcher Kaiming He) accounts for ReLU's behavior by using a different variance scaling:
For a layer with n_in inputs:
- Xavier uses:
Variance = 1 / n_in - He uses:
Variance = 2 / n_in
The factor of 2 compensates for ReLU killing half the gradients! By starting with roughly twice the variance, the network maintains healthy signal flow even after ReLU's zeroing effect.
When to Use He Initialization
Use He initialization whenever you're working with:
- ReLU activation functions
- Leaky ReLU, PReLU, or ELU variants
- Any activation that zeros out negative inputs
Stick with Xavier when using sigmoid, tanh, or other symmetric, zero-centered activations.
Key Takeaway: He initialization uses Variance = 2/n_in instead of Xavier's 1/n_in to compensate for ReLU cutting off half the signal, ensuring proper variance preservation through deep ReLU networks.