Course contentsShow
← Machine Learning and Deep Learning
Lesson 695 of 3,53817. Optimization for Deep LearningFree lesson

Adam: Combining Momentum and Adaptation

Unifying momentum and adaptive learning rates with bias correction for the first moments.

Adam: Combining Momentum and Adaptation

What you'll learn: How the Adam optimizer unifies momentum and adaptive learning rates with bias correction to create one of the most effective training algorithms.

The Marriage of Two Great Ideas

You've learned SGD with momentum (concepts 688-690) that accumulates gradient history for smoother updates, and adaptive methods like RMSprop (694) that scale learning rates per parameter. Adam (Adaptive Moment Estimation) combines both approaches into a single, powerful optimizer.

How Adam Works

Adam maintains two moving averages for each parameter:

  1. First moment (m): An exponentially decaying average of past gradients (like momentum)
  2. Second moment (v): An exponentially decaying average of past squared gradients (like RMSprop)

But there's a critical addition: bias correction. In the early iterations, both m and v start at zero and are biased toward zero. Adam corrects this by dividing by (1 - β₁ᵗ) for the first moment and (1 - β₂ᵗ) for the second moment, where t is the iteration number.

Why Bias Correction Matters

Without bias correction, the optimizer would take unreliably small steps in the first few iterations, since the moving averages haven't "warmed up" yet. Think of it like judging a runner's average speed after only their first step—you'd underestimate badly! Bias correction essentially says "adjust for the fact that we haven't seen many gradients yet."

The Result

Adam gets momentum's smooth traversal of flat regions AND adaptive learning rates that handle different parameter scales, PLUS it starts training effectively from iteration one thanks to bias correction.

Key Takeaway: Adam combines momentum's velocity with RMSprop's adaptive scaling and adds bias correction to the first moments, making it robust and effective from the very first training step.