SGD with Momentum: Mathematics
What you'll learn: How the exponential moving average formulation works mathematically and how the momentum coefficient controls gradient acceleration.
The Core Mathematical Formulation
In vanilla SGD, we update weights directly using the gradient. With momentum, we introduce a velocity term that accumulates gradients over time using an exponential moving average:
v_t = β * v_{t-1} + (1 - β) * g_t
θ_t = θ_{t-1} - α * v_t
Where:
v_tis the velocity at time step tβis the momentum coefficient (typically 0.9 or 0.99)g_tis the current gradientαis the learning rateθ_trepresents the parameters
Understanding the Exponential Moving Average
The momentum term β controls how much history we preserve. Think of it like a heavy ball rolling downhill—it doesn't change direction instantly but builds up speed in consistent directions.
When β = 0.9, each gradient contributes:
- 90% from previous velocity
- 10% from current gradient
Expanding the recursion shows that gradient g_{t-k} is weighted by β^k, creating exponentially decaying influence from past gradients.
How Momentum Controls Acceleration
Higher β (e.g., 0.99): More memory of past gradients, smoother trajectory, stronger acceleration in consistent directions, but slower to change course.
Lower β (e.g., 0.5): Less memory, more responsive to recent gradients, less smoothing, weaker acceleration.
The effective "window" of averaging is approximately 1/(1-β) time steps—so β=0.9 averages over roughly 10 steps, while β=0.99 averages over 100 steps.
Key Takeaway: Momentum uses an exponential moving average where the coefficient β controls how much gradient history accelerates updates—higher β means stronger acceleration in consistent directions but slower adaptation to changes.