Course contentsShow
← Machine Learning and Deep Learning
Lesson 705 of 3,53817. Optimization for Deep LearningFree lesson

Adam: Combining Momentum and Adaptive Rates

Learn how Adam combines momentum with per-parameter learning rates using first and second moment estimates.

Adam: Combining Momentum and Adaptive Rates

What you'll learn: How Adam optimizer unifies momentum's acceleration with per-parameter adaptive learning rates for robust, efficient training.

The Best of Both Worlds

You've seen momentum smooth out gradient updates and RMSprop adapt learning rates per parameter. Adam (Adaptive Moment Estimation) brilliantly combines both strategies into one powerful optimizer.

Adam maintains two moving averages for each parameter:

  1. First moment estimate (m): An exponentially decaying average of past gradients (like momentum)
  2. Second moment estimate (v): An exponentially decaying average of past squared gradients (like RMSprop)

How Adam Works

Think of Adam as a smart driver who:

  • Momentum component (m): Remembers which direction you've been traveling to maintain speed
  • Adaptive component (v): Adjusts the gas pedal differently for each wheel based on how bumpy the terrain has been

The update rule combines these:

  • Compute m = β₁ * m + (1 - β₁) * gradient (momentum-like term)
  • Compute v = β₂ * v + (1 - β₂) * gradient² (RMSprop-like term)
  • Apply bias correction (because m and v start at zero)
  • Update: parameter -= learning_rate * m̂ / (√v̂ + ε)

Why This Combination Matters

Adam adapts the learning rate for each parameter while maintaining directional momentum. Parameters with consistently large gradients get smaller effective learning rates (via large v), while momentum helps traverse flat regions and navigate ravines efficiently.

The result? Fast convergence with less sensitivity to learning rate choice and stable training across diverse architectures.

Key Takeaway: Adam combines momentum's directional acceleration with RMSprop's per-parameter adaptive learning rates, creating a robust optimizer that works well across many problems with minimal tuning.