Course contentsShow
← Machine Learning and Deep Learning
Lesson 706 of 3,53817. Optimization for Deep LearningFree lesson

Adam's Bias Correction Mechanism

Understand why bias correction is needed in early training steps to prevent underestimation of moment estimates.

Adam's Bias Correction Mechanism

What you'll learn: Why Adam applies bias correction to its moment estimates during the early stages of training to prevent systematic underestimation.

The Problem with Exponential Moving Averages

Recall that Adam maintains two exponential moving averages: one for gradients (first moment, m) and one for squared gradients (second moment, v). Both are initialized to zero.

Here's the issue: when you start with zeros and use exponential averaging, your early estimates are biased toward zero. This means in the first few iterations, your moment estimates are much smaller than they should be, leading to overly cautious (tiny) parameter updates right when exploration matters most.

Why This Happens

Think of it like a thermometer that starts at absolute zero. Even after reading a few warm temperatures, the average will still be pulled down by that initial zero reading. The exponential decay weights (β₁ and β₂, typically 0.9 and 0.999) make this worse because they heavily favor recent history—but when "history" includes that initial zero, your estimates lag behind reality.

The Correction Formula

Adam corrects this by dividing the moment estimates by a factor that accounts for how many steps have passed:

  • Corrected first moment: m̂ = m / (1 - β₁ᵗ)
  • Corrected second moment: v̂ = v / (1 - β₂ᵗ)

where t is the current time step.

In early steps, (1 - β₁ᵗ) is small, so dividing by it scales up the estimates. As training progresses and t grows large, β₁ᵗ approaches zero, making the correction factor approach 1—meaning no correction is needed anymore.

Key Takeaway: Adam's bias correction prevents underestimating moment statistics during early training by compensating for zero-initialization, ensuring more reliable parameter updates from the very first steps.