Adam: Combining Momentum and Adaptation
What you'll learn: How the Adam optimizer unifies momentum and adaptive learning rates with bias correction to create one of the most effective training algorithms.
The Marriage of Two Great Ideas
You've learned SGD with momentum (concepts 688-690) that accumulates gradient history for smoother updates, and adaptive methods like RMSprop (694) that scale learning rates per parameter. Adam (Adaptive Moment Estimation) combines both approaches into a single, powerful optimizer.
How Adam Works
Adam maintains two moving averages for each parameter:
- First moment (m): An exponentially decaying average of past gradients (like momentum)
- Second moment (v): An exponentially decaying average of past squared gradients (like RMSprop)
But there's a critical addition: bias correction. In the early iterations, both m and v start at zero and are biased toward zero. Adam corrects this by dividing by (1 - β₁ᵗ) for the first moment and (1 - β₂ᵗ) for the second moment, where t is the iteration number.
Why Bias Correction Matters
Without bias correction, the optimizer would take unreliably small steps in the first few iterations, since the moving averages haven't "warmed up" yet. Think of it like judging a runner's average speed after only their first step—you'd underestimate badly! Bias correction essentially says "adjust for the fact that we haven't seen many gradients yet."
The Result
Adam gets momentum's smooth traversal of flat regions AND adaptive learning rates that handle different parameter scales, PLUS it starts training effectively from iteration one thanks to bias correction.
Key Takeaway: Adam combines momentum's velocity with RMSprop's adaptive scaling and adds bias correction to the first moments, making it robust and effective from the very first training step.