Course contentsShow
← Machine Learning and Deep Learning
Lesson 110 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Constrained Optimization and Lagrange Multipliers

Using Lagrange multipliers to convert equality-constrained problems into unconstrained ones.

Constrained Optimization and Lagrange Multipliers

What You'll Learn

You'll discover how Lagrange multipliers transform equality-constrained optimization problems into unconstrained ones that we can solve using techniques like gradient descent.

The Problem with Constraints

Imagine you want to minimize a function f(x, y), but you must satisfy an equality constraint like g(x, y) = 0 (perhaps x + y = 10). You can't just follow the gradient of f freely—you're stuck on the constraint curve!

The Lagrange Multiplier Trick

Instead of directly solving the constrained problem, we create a new function called the Lagrangian:

L(x, y, λ) = f(x, y) + λ · g(x, y)

Here, λ (lambda) is the Lagrange multiplier—a new variable that "enforces" the constraint. Now we have an unconstrained problem: find where the gradient of L equals zero.

Why This Works

Think of λ as a penalty weight. When we minimize L, we're balancing two goals:

  1. Minimize the original function f
  2. Keep the constraint g = 0 satisfied

At the optimal point, the gradient of f and the gradient of g must be parallel (pointing in the same or opposite directions). The multiplier λ captures how much the constraint is "pulling" the solution away from the unconstrained optimum.

The Mechanics

To solve:

  1. Form the Lagrangian: L(x, λ) = f(x) + λ · g(x)
  2. Take partial derivatives: ∂L/∂x = 0 and ∂L/∂λ = 0
  3. Solve the system of equations

The second condition (∂L/∂λ = 0) recovers your original constraint g(x) = 0.

Key Takeaway: Lagrange multipliers convert constrained optimization into unconstrained optimization by introducing extra variables that encode the constraints, enabling us to use standard gradient-based methods.