Course contentsShow
← Machine Learning and Deep Learning
Lesson 688 of 3,53817. Optimization for Deep LearningFree lesson

SGD with Momentum: Concept

Adding velocity to gradient descent to accelerate convergence and dampen oscillations in ravines.

SGD with Momentum: Concept

What you'll learn: How momentum helps gradient descent converge faster and smoother by accumulating past gradients like a rolling ball gains speed downhill.

The Problem with Basic SGD

Standard SGD (Stochastic Gradient Descent) updates weights based solely on the current gradient. This creates problems:

  • Oscillation in ravines: When the loss surface has steep slopes in some directions and gentle slopes in others (like a narrow valley), SGD zigzags back and forth, making slow progress toward the minimum.
  • Slow convergence: Each step "forgets" previous directions, missing opportunities to build up speed in consistent directions.

The Momentum Solution

Momentum adds a velocity term that accumulates gradients over time. Think of it like a ball rolling downhill:

  • On a flat slope, the ball gradually speeds up
  • In a ravine, sideways oscillations cancel out while forward motion accumulates
  • The ball doesn't stop instantly when hitting a small bump

Mathematically, momentum introduces a velocity vector v that combines:

  1. The current gradient (where you should go now)
  2. The previous velocity (where you've been going)

The velocity update becomes:

v = β * v - learning_rate * gradient
weight = weight + v

Where β (beta, typically 0.9) is the momentum coefficient controlling how much past gradients influence the current update.

Why It Works

Momentum dampens oscillations because opposing gradients cancel out in the velocity, while consistent gradients accumulate. This accelerates convergence in the true descent direction while reducing zigzagging.

Key Takeaway: Momentum transforms SGD from a memoryless process into one with directional memory, using exponentially-weighted averages of past gradients to accelerate convergence and smooth the optimization path through ravines.