Course contentsShow
← Machine Learning and Deep Learning
Lesson 45 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Critical Points and Extrema

Finding local maxima, minima, and saddle points by setting gradients to zero.

Critical Points and Extrema

What you'll learn: How to identify potential maxima, minima, and saddle points by finding where the gradient equals zero.

The Core Idea

Imagine you're standing on a hilly landscape. The critical points are locations where the ground is perfectly flat in all directions—you could place a marble there and it wouldn't roll anywhere. These occur where the gradient vector equals zero.

In machine learning, we constantly search for minima (lowest points) of loss functions to optimize our models. Critical points are our candidates for these optimal locations.

Finding Critical Points

Since the gradient vector points in the direction of steepest ascent, when it equals zero, we've found a spot where the function isn't increasing or decreasing in any direction.

For a function f(x₁, x₂, ..., xₙ), set all partial derivatives to zero:

∂f/∂x₁ = 0
∂f/∂x₂ = 0
...
∂f/∂xₙ = 0

Solving this system of equations gives you the critical points.

Three Types of Critical Points

  1. Local Minimum: The function value is lowest in the surrounding neighborhood (a valley)
  2. Local Maximum: The function value is highest nearby (a hilltop)
  3. Saddle Point: A minimum in some directions but a maximum in others (like a mountain pass)

In deep learning, saddle points are surprisingly common in high-dimensional spaces and can slow down optimization algorithms.

Why This Matters

When training neural networks, gradient descent searches for critical points where the loss is minimized. Understanding that setting gradients to zero identifies candidates for optimization helps you grasp why algorithms check these locations.

Key Takeaway: Critical points occur where all partial derivatives equal zero; they may be local maxima, minima, or saddle points—you need additional tests (like the Hessian matrix) to determine which type you've found.