Course contentsShow
← Machine Learning and Deep Learning
Lesson 668 of 3,53816. Activation Functions and Weight InitializationFree lesson

Xavier/Glorot Initialization

Understanding Xavier initialization for sigmoid/tanh networks and its variance-preserving properties.

Xavier/Glorot Initialization

What you'll learn: How Xavier initialization keeps signal strength balanced throughout sigmoid and tanh networks by carefully choosing initial weight scales.


The Problem Xavier Solves

Remember the variance preservation principle? When signals pass through layers, we want to maintain consistent variance so gradients neither vanish nor explode. Xavier initialization (named after Xavier Glorot) provides a mathematically principled way to set initial weights for networks using sigmoid or tanh activations.

How Xavier Initialization Works

Xavier initialization sets weights by sampling from a distribution with variance:

Variance = 2 / (n_in + n_out)

Where n_in is the number of input neurons and n_out is the number of output neurons for that layer.

Typically implemented as:

  • Uniform distribution: sample from [-limit, +limit] where limit = √(6 / (n_in + n_out))
  • Normal distribution: sample from N(0, 2/(n_in + n_out))

Why This Specific Formula?

Xavier analyzed the linear regime of sigmoid and tanh functions (near zero, where they behave almost linearly). The formula balances:

  • Forward pass: Keeps activation variance stable as signals flow forward
  • Backward pass: Keeps gradient variance stable as errors flow backward

The averaging of n_in and n_out creates a compromise that works well in both directions.

When to Use Xavier

Use Xavier initialization specifically for:

  • Networks with sigmoid activations
  • Networks with tanh activations

It assumes symmetric, near-linear activation behavior around zero—which sigmoid and tanh provide.

Key Takeaway: Xavier/Glorot initialization preserves variance in both forward and backward passes for sigmoid/tanh networks by scaling initial weights based on layer dimensions, preventing vanishing/exploding gradients from the start.