Momentum Methods
What You'll Learn
You'll learn how momentum speeds up gradient descent by using accumulated information from previous update steps.
The Core Idea
Think of gradient descent like rolling a ball downhill. Without momentum, the ball stops and restarts with every step based only on the current slope. Momentum methods remember which direction the ball was already moving and keep it going in that direction, making progress smoother and faster.
In standard gradient descent, each update depends only on the current gradient. Momentum introduces a "velocity" term that accumulates past gradients with exponential decay:
v_t = β * v_(t-1) + gradient_t
parameter_t+1 = parameter_t - learning_rate * v_t
Here, β (typically 0.9) controls how much past information to retain. A higher β means more momentum—the optimizer trusts the accumulated direction more than the immediate gradient.
Why Momentum Helps
-
Smooths noisy gradients: In stochastic gradient descent, individual batch gradients can be noisy. Momentum averages over recent history, reducing oscillation.
-
Accelerates progress in consistent directions: If gradients repeatedly point the same way (like in a long valley), momentum builds up speed in that direction.
-
Dampens oscillations: In narrow valleys where gradients alternate directions, momentum prevents the optimizer from bouncing back and forth.
The Intuition
Imagine pushing a heavy cart. Each push (gradient) adds to its existing motion (velocity). The cart doesn't instantly change direction with every small push—its accumulated momentum keeps it moving steadily toward the bottom.
Key Takeaway: Momentum methods accelerate gradient descent by accumulating an exponentially weighted average of past gradients, smoothing updates and speeding convergence, especially in stochastic settings or along consistent descent directions.