Nesterov Accelerated Gradient
What you'll learn: How Nesterov momentum "looks ahead" before calculating gradients to achieve faster, smarter convergence than standard momentum.
The Core Insight
Remember standard momentum from the previous lesson? It accumulates past gradients into a velocity term and moves in that direction. Nesterov Accelerated Gradient (NAG) adds a clever twist: instead of computing the gradient at your current position, you first "jump" to where momentum would take you, then compute the gradient there.
Think of it like this: Standard momentum is like a ball rolling downhill that picks up speed but might overshoot the valley. Nesterov momentum is like a smart ball that looks ahead to where it's about to land, checks the slope there, and adjusts its trajectory accordingly before overshooting.
The Mathematical Difference
Standard momentum updates:
- Compute gradient at current position:
g_t = ∇L(θ_t) - Update velocity:
v_t = β·v_{t-1} + g_t - Move:
θ_{t+1} = θ_t - α·v_t
Nesterov momentum updates:
- Look ahead first:
θ_lookahead = θ_t - β·v_{t-1} - Compute gradient at lookahead position:
g_t = ∇L(θ_lookahead) - Update velocity:
v_t = β·v_{t-1} + g_t - Move:
θ_{t+1} = θ_t - α·v_t
The key is step 1: we evaluate the gradient not where we are, but where momentum would take us. This "anticipatory" gradient gives better correction when approaching minima.
Why It Works Better
By looking ahead, NAG gets advance warning when it's about to overshoot. It can slow down more intelligently near optimal points, leading to faster convergence and less oscillation around minima.
Key Takeaway: Nesterov Accelerated Gradient improves upon standard momentum by computing gradients at the anticipated future position rather than the current position, enabling smarter corrections and faster convergence.