Course contentsShow
← Machine Learning and Deep Learning
Lesson 102 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Convergence Guarantees for Gradient Descent

Conditions under which gradient descent converges to global or local minima, focusing on convex cases.

Convergence Guarantees for Gradient Descent

What You'll Learn

You'll understand when and why gradient descent is guaranteed to reach an optimal solution.

The Core Idea

Gradient descent doesn't always promise to find the best answer—but under certain conditions, it does! The most important condition is convexity. When your function is convex (bowl-shaped with no bumps or valleys inside), gradient descent with an appropriate learning rate will always converge to the global minimum—the absolute best solution.

Think of it like rolling a ball down a smooth bowl. No matter where you drop it, it'll eventually settle at the bottom. That's a convex function. But if the landscape has multiple valleys (non-convex), the ball might get stuck in a small dip that isn't the deepest point—a local minimum.

Key Conditions for Convergence

For convex functions, gradient descent converges to the global minimum if:

  • The learning rate is small enough (not overshooting)
  • The learning rate satisfies certain bounds related to the function's smoothness
  • You run enough iterations

For strongly convex functions (even stricter bowl shape), convergence is faster and more predictable—you get exponential convergence rates.

For non-convex functions (most real deep learning problems), gradient descent only guarantees convergence to a local minimum or critical point where the gradient is zero. There's no guarantee it's the global best.

Why This Matters

Understanding convergence guarantees helps you:

  • Know when gradient descent is trustworthy (convex problems)
  • Set realistic expectations for deep learning (non-convex)
  • Choose appropriate learning rates
  • Decide if you need more sophisticated optimization methods

Key Takeaway: Gradient descent converges to the global minimum for convex functions with proper learning rates, but only finds local minima in non-convex cases—which is why deep learning often requires careful initialization and advanced variants.