Bias Initialization
What you'll learn: How to properly initialize bias terms in neural networks to support effective training from the start.
While weight initialization requires careful variance control to preserve signal flow, bias initialization is usually much simpler—but still important to get right.
The Standard Approach: Zero Initialization
For most cases, biases are initialized to zero. Unlike weights, setting all biases to the same value doesn't break symmetry because:
- Weights already break symmetry — different random weights ensure neurons learn different features
- Biases shift activations — they offset the weighted sum, allowing the network to center its activations appropriately during training
- Zero is neutral — starting at zero lets the network learn positive or negative offsets as needed
Think of bias as a "starting volume knob" for each neuron. Setting it to zero means "neutral position," and training will turn it up or down based on what the data requires.
Special Cases: When NOT to Use Zero
Some architectural components benefit from non-zero bias initialization:
- Forget gates in LSTMs: Initialize biases to small positive values (e.g., 1.0 or 2.0) to make the network initially "remember" information rather than forget it
- Output layers with class imbalance: Initialize biases to reflect prior class probabilities (e.g., log of class frequencies)
- Batch normalization layers: Biases are typically initialized to zero, but the scale parameter may start at one
Why Zero Works
Since biases add a constant to pre-activations, they don't multiply signals like weights do. Zero initialization:
- Doesn't cause vanishing or exploding gradients
- Allows each neuron to learn its optimal offset independently
- Works consistently across different activation functions
Key Takeaway: Initialize biases to zero in most cases—it's simple, safe, and effective. Use non-zero initialization only for special architectural components like LSTM forget gates or when domain knowledge suggests a better starting point.