Course contentsShow
← Machine Learning and Deep Learning
Lesson 95 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Local vs Global Optima

Understanding local minima, maxima, saddle points, and global optima in optimization landscapes.

Local vs Global Optima

You'll learn how optimization landscapes contain multiple peaks and valleys, and why finding the absolute best solution isn't always straightforward.

The Landscape Analogy

Imagine you're hiking in foggy mountains, trying to reach the highest peak. You climb upward until you can't go any higher—but is this the tallest mountain in the range, or just a local hilltop? In optimization, this dilemma appears constantly.

Core Concepts

When optimizing a function (like minimizing a loss function in ML), you encounter different types of points:

Local Minimum: A point where the function value is lower than all nearby points, but not necessarily the lowest overall. Think of it as a valley surrounded by hills.

Global Minimum: The absolute lowest point across the entire function—the deepest valley in the entire landscape.

Local Maximum: A point higher than all nearby points. Picture a hilltop.

Global Maximum: The absolute highest point everywhere.

Saddle Point: A point that's a minimum in some directions but a maximum in others—like a mountain pass or a horse saddle.

Why This Matters

Most ML algorithms use iterative optimization (like gradient descent) that moves "downhill" step by step. This strategy can get trapped in local minima, never discovering the global minimum. The algorithm only sees its immediate neighborhood, not the entire landscape.

For complex models with millions of parameters, the optimization landscape becomes extremely high-dimensional with countless local optima. Understanding this helps you choose appropriate optimization strategies and interpret your model's behavior.

Key Takeaway: Local optima are points that appear optimal in their neighborhood, while global optima are the absolute best solution—optimization algorithms must navigate carefully to avoid getting stuck in local traps.