Nesterov Accelerated Gradient
What you'll learn: How Nesterov momentum's "lookahead" mechanism provides smarter gradient estimates for faster convergence.
The Lookahead Insight
Remember momentum from lesson 688? It accumulates past gradients to build velocity, like a ball rolling downhill. Nesterov Accelerated Gradient (NAG) adds a clever twist: instead of computing the gradient at your current position, you compute it at an approximate future position where momentum would take you next.
Think of it like this: standard momentum is like running blindfolded while remembering which direction you've been moving. Nesterov momentum is like peeking ahead slightly before deciding your next step.
How It Works
The key difference happens in two stages:
- Lookahead step: First, use your current momentum to jump to an intermediate position (without updating weights yet)
- Corrected gradient: Compute the gradient at that lookahead position
- Update: Use this "future-informed" gradient to update both velocity and weights
Mathematically, where standard momentum does:
- v = β·v - α·∇L(θ)
- θ = θ + v
Nesterov does:
- v = β·v - α·∇L(θ + β·v) ← gradient evaluated ahead
- θ = θ + v
Why This Helps
By evaluating the gradient where momentum is about to take you, NAG gets better information. If momentum is carrying you toward a mistake, the lookahead gradient can "apply the brakes" sooner. This reduces overshooting and oscillation, leading to faster, more stable convergence.
Key Takeaway: Nesterov Accelerated Gradient improves upon standard momentum by computing gradients at the anticipated future position rather than the current one, enabling more informed updates and faster convergence with less overshooting.