Critical Points and Extrema
What you'll learn: How to identify potential maxima, minima, and saddle points by finding where the gradient equals zero.
The Core Idea
Imagine you're standing on a hilly landscape. The critical points are locations where the ground is perfectly flat in all directions—you could place a marble there and it wouldn't roll anywhere. These occur where the gradient vector equals zero.
In machine learning, we constantly search for minima (lowest points) of loss functions to optimize our models. Critical points are our candidates for these optimal locations.
Finding Critical Points
Since the gradient vector points in the direction of steepest ascent, when it equals zero, we've found a spot where the function isn't increasing or decreasing in any direction.
For a function f(x₁, x₂, ..., xₙ), set all partial derivatives to zero:
∂f/∂x₁ = 0
∂f/∂x₂ = 0
...
∂f/∂xₙ = 0
Solving this system of equations gives you the critical points.
Three Types of Critical Points
- Local Minimum: The function value is lowest in the surrounding neighborhood (a valley)
- Local Maximum: The function value is highest nearby (a hilltop)
- Saddle Point: A minimum in some directions but a maximum in others (like a mountain pass)
In deep learning, saddle points are surprisingly common in high-dimensional spaces and can slow down optimization algorithms.
Why This Matters
When training neural networks, gradient descent searches for critical points where the loss is minimized. Understanding that setting gradients to zero identifies candidates for optimization helps you grasp why algorithms check these locations.
Key Takeaway: Critical points occur where all partial derivatives equal zero; they may be local maxima, minima, or saddle points—you need additional tests (like the Hessian matrix) to determine which type you've found.