Course contentsShow
← Machine Learning and Deep Learning
Lesson 99 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Second-Order Optimality Conditions

Using the Hessian matrix to verify local minima, maxima, and saddle points via second derivatives.

Second-Order Optimality Conditions

What you'll learn: How to use the Hessian matrix and second derivatives to confirm whether a critical point is a local minimum, maximum, or saddle point.

Why First Derivatives Aren't Enough

When you found critical points using gradients (first derivatives equal to zero), you discovered candidates for optima. But a zero gradient doesn't tell you what kind of point you've found. Is it a valley (minimum), a peak (maximum), or a saddle point that's neither?

This is where second-order optimality conditions come in.

The Hessian Test

Remember the Hessian matrix contains all second partial derivatives of your function. It captures how the slope itself is changing—the "curvature" of your function.

At a critical point, examine the Hessian:

  • Positive definite Hessian (all eigenvalues > 0): The function curves upward in all directions → local minimum ✓
  • Negative definite Hessian (all eigenvalues < 0): The function curves downward in all directions → local maximum
  • Indefinite Hessian (mixed positive and negative eigenvalues): Curves up in some directions, down in others → saddle point
  • Positive or negative semidefinite (some eigenvalues = 0): The test is inconclusive

Intuition

Think of a bowl (minimum), an upside-down bowl (maximum), or a mountain pass (saddle). The Hessian's eigenvalues tell you whether the surface curves consistently in one direction or has mixed curvature.

In deep learning, saddle points are common in high-dimensional loss landscapes. Identifying them helps explain why certain optimization algorithms struggle or succeed.

Key Takeaway: The Hessian matrix at a critical point determines its nature—positive definite means local minimum, negative definite means maximum, and mixed signs indicate a saddle point.