Constrained Optimization and Lagrange Multipliers
What You'll Learn
You'll discover how Lagrange multipliers transform equality-constrained optimization problems into unconstrained ones that we can solve using techniques like gradient descent.
The Problem with Constraints
Imagine you want to minimize a function f(x, y), but you must satisfy an equality constraint like g(x, y) = 0 (perhaps x + y = 10). You can't just follow the gradient of f freely—you're stuck on the constraint curve!
The Lagrange Multiplier Trick
Instead of directly solving the constrained problem, we create a new function called the Lagrangian:
L(x, y, λ) = f(x, y) + λ · g(x, y)
Here, λ (lambda) is the Lagrange multiplier—a new variable that "enforces" the constraint. Now we have an unconstrained problem: find where the gradient of L equals zero.
Why This Works
Think of λ as a penalty weight. When we minimize L, we're balancing two goals:
- Minimize the original function
f - Keep the constraint
g = 0satisfied
At the optimal point, the gradient of f and the gradient of g must be parallel (pointing in the same or opposite directions). The multiplier λ captures how much the constraint is "pulling" the solution away from the unconstrained optimum.
The Mechanics
To solve:
- Form the Lagrangian:
L(x, λ) = f(x) + λ · g(x) - Take partial derivatives:
∂L/∂x = 0and∂L/∂λ = 0 - Solve the system of equations
The second condition (∂L/∂λ = 0) recovers your original constraint g(x) = 0.
Key Takeaway: Lagrange multipliers convert constrained optimization into unconstrained optimization by introducing extra variables that encode the constraints, enabling us to use standard gradient-based methods.