Course contentsShow
← Machine Learning and Deep Learning
Lesson 690 of 3,53817. Optimization for Deep LearningFree lesson

Nesterov Accelerated Gradient

Looking ahead before computing gradients to achieve faster convergence than classical momentum.

Nesterov Accelerated Gradient

What you'll learn: How Nesterov momentum "looks ahead" before calculating gradients to achieve faster, smarter convergence than standard momentum.

The Core Insight

Remember standard momentum from the previous lesson? It accumulates past gradients into a velocity term and moves in that direction. Nesterov Accelerated Gradient (NAG) adds a clever twist: instead of computing the gradient at your current position, you first "jump" to where momentum would take you, then compute the gradient there.

Think of it like this: Standard momentum is like a ball rolling downhill that picks up speed but might overshoot the valley. Nesterov momentum is like a smart ball that looks ahead to where it's about to land, checks the slope there, and adjusts its trajectory accordingly before overshooting.

The Mathematical Difference

Standard momentum updates:

  1. Compute gradient at current position: g_t = ∇L(θ_t)
  2. Update velocity: v_t = β·v_{t-1} + g_t
  3. Move: θ_{t+1} = θ_t - α·v_t

Nesterov momentum updates:

  1. Look ahead first: θ_lookahead = θ_t - β·v_{t-1}
  2. Compute gradient at lookahead position: g_t = ∇L(θ_lookahead)
  3. Update velocity: v_t = β·v_{t-1} + g_t
  4. Move: θ_{t+1} = θ_t - α·v_t

The key is step 1: we evaluate the gradient not where we are, but where momentum would take us. This "anticipatory" gradient gives better correction when approaching minima.

Why It Works Better

By looking ahead, NAG gets advance warning when it's about to overshoot. It can slow down more intelligently near optimal points, leading to faster convergence and less oscillation around minima.

Key Takeaway: Nesterov Accelerated Gradient improves upon standard momentum by computing gradients at the anticipated future position rather than the current position, enabling smarter corrections and faster convergence.