Course contentsShow
← Machine Learning and Deep Learning
Lesson 94 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Unconstrained vs Constrained Optimization

Distinguish between optimization with and without constraints, and when each type appears in ML.

Unconstrained vs Constrained Optimization

What you'll learn: You'll understand the difference between optimization problems with and without restrictions, and recognize where each appears in machine learning.

The Core Distinction

In unconstrained optimization, you're free to search for the best solution anywhere—no boundaries, no rules. You simply want to minimize or maximize your objective function over all possible values.

In constrained optimization, you must find the best solution while respecting certain limitations. These constraints might be equalities (e.g., "weights must sum to 1") or inequalities (e.g., "all values must be non-negative").

An Everyday Analogy

Think of planning a vacation:

  • Unconstrained: Find the absolute best destination in the world, regardless of cost or travel time
  • Constrained: Find the best destination you can afford with your $2000 budget and 5 vacation days

When Each Appears in ML

Unconstrained optimization dominates basic machine learning:

  • Training standard neural networks (minimizing loss with gradient descent)
  • Linear regression (finding optimal weights)
  • Most deep learning scenarios where parameters can take any real value

Constrained optimization appears when:

  • Support Vector Machines (margin constraints)
  • Portfolio optimization (budget constraints, non-negativity)
  • Regularization with explicit parameter bounds
  • Fairness constraints (ensuring algorithmic fairness)
  • Physical constraints (probabilities must sum to 1)

Unconstrained problems are generally simpler—you just follow the gradient downhill. Constrained problems require specialized techniques to stay within the allowed region while optimizing.

Key Takeaway: Unconstrained optimization lets you search freely (common in neural network training), while constrained optimization requires respecting boundaries or rules (common in SVMs and specialized ML applications).