Course contentsShow
← Machine Learning and Deep Learning
Lesson 98 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

First-Order Optimality Conditions

Using gradients to identify optima: zero gradient at minima for unconstrained smooth problems.

First-Order Optimality Conditions

What you'll learn: How to use gradients to mathematically identify where a smooth function reaches its minimum or maximum.

The Core Idea

When you're trying to minimize a smooth function (like a loss function in machine learning), you need a way to know when you've reached the bottom. The first-order optimality condition gives you exactly that: at a minimum, the gradient must be zero.

Think of it like this: imagine you're standing in a valley. The gradient tells you which direction slopes downward. If there's any downward direction available, you're not at the lowest point yet. Only when every direction around you is flat or upward—meaning the gradient is zero—have you potentially found a minimum.

Why Zero Gradient?

The gradient vector (which you learned earlier) points in the direction of steepest ascent. If the gradient isn't zero, you can move in the opposite direction to decrease the function value. Therefore, at any local minimum of an unconstrained, smooth optimization problem:

∇f(x) = 0*

where x* is the optimal point and ∇f is the gradient.

Important Note

A zero gradient is a necessary condition for a minimum, but not sufficient—it could also indicate a maximum or a saddle point. That's where the Hessian matrix and second-order tests (which you've already learned) come in to confirm what type of critical point you've found.

Key Takeaway: For smooth unconstrained optimization problems, any local minimum must have a zero gradient; this is the first-order optimality condition and forms the mathematical foundation for gradient-based optimization algorithms in machine learning.