Variance Preservation Principle
What you'll learn: How to maintain consistent signal strength (variance) throughout a deep neural network by properly initializing weights.
The Core Problem
Imagine shouting a message down a long hallway with many doors. If each door dampens the sound too much, by the final room nobody hears you. If each door amplifies it too much, the message becomes deafening noise. The Variance Preservation Principle ensures your neural network's "signal" stays at just the right volume as it passes through each layer.
When data flows forward through layers during training, we want the variance (spread) of activations to remain roughly constant. Why? If variance shrinks layer by layer, deep layers receive vanishingly small signals—they can't learn effectively. If variance grows, activations explode into numerically unstable values.
The same principle applies backward during backpropagation: gradients should maintain consistent variance as they flow from output back to input. If gradients vanish, early layers barely update. If they explode, training becomes chaotic.
The Mathematical Goal
For a layer with n_in input connections, we want:
Forward pass: Var(output) ≈ Var(input)
Backward pass: Var(gradient_in) ≈ Var(gradient_out)
This balance depends on both the weight initialization and the activation function. By deriving what variance our initial weights should have (based on layer size and activation choice), we prevent the signal from degrading or exploding as networks get deeper.
Without this principle, deep networks (10+ layers) would be nearly impossible to train—early researchers hit this wall repeatedly before understanding variance preservation.
Key Takeaway: The Variance Preservation Principle guides weight initialization to maintain stable activation and gradient variance across all layers, enabling successful training of deep networks.