Parallel Gradients at Optimal Points
What You'll Learn
You'll discover why, at a constrained optimum, the gradient of your objective function must be parallel to the gradient of your constraint.
The Core Geometric Insight
Imagine you're hiking on a mountain (your objective function) but you're forced to stay on a specific trail (your constraint). Where is the highest point you can reach on that trail?
Here's the key: at that optimal point, you can't move along the trail in a way that increases your altitude. If you could, you wouldn't be at a maximum!
Now think about what the gradient vectors mean:
- ∇f points in the direction of steepest ascent for your objective function
- ∇g is perpendicular to the constraint curve (the trail)
At the constrained maximum, any small move along the trail must not increase f. But directions along the trail are exactly those perpendicular to ∇g. If ∇f had any component perpendicular to ∇g, you could move that way along the constraint and change f—contradicting that you're at an optimum.
Therefore, ∇f must have no component perpendicular to ∇g, which means ∇f and ∇g must be parallel (or anti-parallel). Mathematically:
∇f = λ∇g
for some scalar λ (called the Lagrange multiplier).
Why This Matters
This geometric fact is the foundation of the method of Lagrange multipliers. It transforms a constrained optimization problem into a system of equations you can solve algebraically.
Key Takeaway: At a constrained extremum, the gradients of the objective and constraint are parallel because any movement along the constraint must not change the objective value—which is only possible when ∇f has no component tangent to the constraint.