Course contentsShow
← Machine Learning and Deep Learning
Lesson 700 of 3,53817. Optimization for Deep LearningFree lesson

Momentum-Based Optimization

Learn how momentum accumulates gradients over time to accelerate convergence and smooth optimization trajectories.

Momentum-Based Optimization

What you'll learn: How momentum accumulates gradients over time to accelerate convergence and create smoother optimization paths through the loss landscape.

The Core Idea

Imagine pushing a ball down a hilly terrain toward the lowest valley. Without momentum, the ball moves purely based on the current slope—it might get stuck in small dips or zigzag inefficiently. With momentum, the ball accumulates velocity from previous movements, allowing it to roll through small bumps and move faster in consistent directions.

Momentum-based optimization applies this physics analogy to gradient descent. Instead of updating weights based solely on the current gradient, momentum maintains a "velocity" term that accumulates past gradients. This creates two powerful effects:

  1. Acceleration in consistent directions: When gradients point the same way across multiple steps, momentum builds up speed in that direction
  2. Smoothing in oscillating directions: When gradients oscillate (like in narrow valleys), momentum dampens the zigzagging by averaging them out

How It Works

At each training step, momentum updates a velocity vector (not just the weights). This velocity is a weighted combination of the previous velocity and the current gradient. The weights then move according to this accumulated velocity rather than the raw gradient alone.

The result? Faster convergence in shallow, consistent regions and more stable progress through areas where standard SGD would bounce around wastefully.

Why It Matters

Without momentum, SGD can be painfully slow in regions where the loss surface is elongated or has different curvatures in different directions. Momentum helps your optimizer "remember" where it was heading and maintain that general direction, dramatically speeding up training in many deep learning scenarios.

Key Takeaway: Momentum transforms optimization from reactive (following only the current gradient) to forward-looking (building velocity from gradient history), enabling faster and smoother convergence through the loss landscape.