Course contentsShow
← Machine Learning and Deep Learning
Lesson 689 of 3,53817. Optimization for Deep LearningFree lesson

SGD with Momentum: Mathematics

The exponential moving average formulation and how the momentum coefficient controls acceleration.

SGD with Momentum: Mathematics

What you'll learn: How the exponential moving average formulation works mathematically and how the momentum coefficient controls gradient acceleration.

The Core Mathematical Formulation

In vanilla SGD, we update weights directly using the gradient. With momentum, we introduce a velocity term that accumulates gradients over time using an exponential moving average:

v_t = β * v_{t-1} + (1 - β) * g_t
θ_t = θ_{t-1} - α * v_t

Where:

  • v_t is the velocity at time step t
  • β is the momentum coefficient (typically 0.9 or 0.99)
  • g_t is the current gradient
  • α is the learning rate
  • θ_t represents the parameters

Understanding the Exponential Moving Average

The momentum term β controls how much history we preserve. Think of it like a heavy ball rolling downhill—it doesn't change direction instantly but builds up speed in consistent directions.

When β = 0.9, each gradient contributes:

  • 90% from previous velocity
  • 10% from current gradient

Expanding the recursion shows that gradient g_{t-k} is weighted by β^k, creating exponentially decaying influence from past gradients.

How Momentum Controls Acceleration

Higher β (e.g., 0.99): More memory of past gradients, smoother trajectory, stronger acceleration in consistent directions, but slower to change course.

Lower β (e.g., 0.5): Less memory, more responsive to recent gradients, less smoothing, weaker acceleration.

The effective "window" of averaging is approximately 1/(1-β) time steps—so β=0.9 averages over roughly 10 steps, while β=0.99 averages over 100 steps.

Key Takeaway: Momentum uses an exponential moving average where the coefficient β controls how much gradient history accelerates updates—higher β means stronger acceleration in consistent directions but slower adaptation to changes.