Local vs Global Optima
You'll learn how optimization landscapes contain multiple peaks and valleys, and why finding the absolute best solution isn't always straightforward.
The Landscape Analogy
Imagine you're hiking in foggy mountains, trying to reach the highest peak. You climb upward until you can't go any higher—but is this the tallest mountain in the range, or just a local hilltop? In optimization, this dilemma appears constantly.
Core Concepts
When optimizing a function (like minimizing a loss function in ML), you encounter different types of points:
Local Minimum: A point where the function value is lower than all nearby points, but not necessarily the lowest overall. Think of it as a valley surrounded by hills.
Global Minimum: The absolute lowest point across the entire function—the deepest valley in the entire landscape.
Local Maximum: A point higher than all nearby points. Picture a hilltop.
Global Maximum: The absolute highest point everywhere.
Saddle Point: A point that's a minimum in some directions but a maximum in others—like a mountain pass or a horse saddle.
Why This Matters
Most ML algorithms use iterative optimization (like gradient descent) that moves "downhill" step by step. This strategy can get trapped in local minima, never discovering the global minimum. The algorithm only sees its immediate neighborhood, not the entire landscape.
For complex models with millions of parameters, the optimization landscape becomes extremely high-dimensional with countless local optima. Understanding this helps you choose appropriate optimization strategies and interpret your model's behavior.
Key Takeaway: Local optima are points that appear optimal in their neighborhood, while global optima are the absolute best solution—optimization algorithms must navigate carefully to avoid getting stuck in local traps.