Second Derivative Test in Multiple Dimensions
What you'll learn: How to use the Hessian matrix to determine whether a critical point is a minimum, maximum, or saddle point in multivariable functions.
The Core Idea
Remember how the second derivative of a single-variable function tells you if a critical point is a minimum (curve opens upward) or maximum (opens downward)? The Hessian matrix does the same job in multiple dimensions, but instead of checking a single value, we examine properties of the entire matrix.
At a critical point (where the gradient is zero), the Hessian matrix acts like a multidimensional "curvature detector":
Classification Rules
- Positive definite Hessian → The function curves upward in all directions → Local minimum
- Negative definite Hessian → The function curves downward in all directions → Local maximum
- Indefinite Hessian → The function curves up in some directions, down in others → Saddle point
- Semidefinite → The test is inconclusive
An Analogy
Imagine standing at a critical point on a landscape. The Hessian tells you about the terrain in every direction around you:
- If all directions slope upward → you're in a valley (minimum)
- If all directions slope downward → you're on a hilltop (maximum)
- If some directions go up while others go down → you're on a mountain pass or saddle
Why This Matters in ML
When training neural networks, we search for minima of loss functions. The Hessian helps us understand whether we've found a good minimum or gotten stuck at a saddle point (common in high-dimensional spaces).
Key Takeaway: The Second Derivative Test uses the definiteness of the Hessian matrix to classify critical points—positive definite means minimum, negative definite means maximum, and indefinite means saddle point.