Course contentsShow
← Machine Learning and Deep Learning
Lesson 671 of 3,53816. Activation Functions and Weight InitializationFree lesson

Bias Initialization

Best practices for initializing bias terms: zero initialization and special cases like forget gates.

Bias Initialization

What you'll learn: How to properly initialize bias terms in neural networks to support effective training from the start.

While weight initialization requires careful variance control to preserve signal flow, bias initialization is usually much simpler—but still important to get right.

The Standard Approach: Zero Initialization

For most cases, biases are initialized to zero. Unlike weights, setting all biases to the same value doesn't break symmetry because:

  1. Weights already break symmetry — different random weights ensure neurons learn different features
  2. Biases shift activations — they offset the weighted sum, allowing the network to center its activations appropriately during training
  3. Zero is neutral — starting at zero lets the network learn positive or negative offsets as needed

Think of bias as a "starting volume knob" for each neuron. Setting it to zero means "neutral position," and training will turn it up or down based on what the data requires.

Special Cases: When NOT to Use Zero

Some architectural components benefit from non-zero bias initialization:

  • Forget gates in LSTMs: Initialize biases to small positive values (e.g., 1.0 or 2.0) to make the network initially "remember" information rather than forget it
  • Output layers with class imbalance: Initialize biases to reflect prior class probabilities (e.g., log of class frequencies)
  • Batch normalization layers: Biases are typically initialized to zero, but the scale parameter may start at one

Why Zero Works

Since biases add a constant to pre-activations, they don't multiply signals like weights do. Zero initialization:

  • Doesn't cause vanishing or exploding gradients
  • Allows each neuron to learn its optimal offset independently
  • Works consistently across different activation functions

Key Takeaway: Initialize biases to zero in most cases—it's simple, safe, and effective. Use non-zero initialization only for special architectural components like LSTM forget gates or when domain knowledge suggests a better starting point.