Dual Formulation and Lagrange Multipliers
You'll learn: How converting the SVM optimization problem into its dual form reveals that only a handful of points—the support vectors—actually matter for making predictions.
Why Go Dual?
So far, you've seen the primal formulation of SVM: minimize the weight vector subject to constraints ensuring correct classification with margin. But there's an equivalent way to solve this problem called the dual formulation, discovered using Lagrange multipliers.
Think of it like solving a puzzle two different ways: one focuses on the puzzle pieces (weights), the other focuses on the relationships between pieces (training points).
The Lagrange Multiplier Approach
When we have a constrained optimization problem (minimize something while satisfying constraints), Lagrange multipliers let us convert it into an unconstrained problem. We introduce a multiplier αᵢ (alpha) for each training point's constraint.
The magic: after the mathematical transformation, the dual problem depends only on:
- The alphas (one per training point)
- Dot products between training points
Most αᵢ turn out to be zero! Only the points lying exactly on or violating the margin have αᵢ > 0—these are your support vectors.
Why This Matters
In the dual form:
- The weight vector w can be written as a combination of only the support vectors
- Predictions depend solely on comparing the new point to support vectors
- All other training points? Completely irrelevant once trained!
This is profound: a dataset with 10,000 points might have only 50 support vectors—those 50 entirely determine the decision boundary.
Key Takeaway: The dual formulation using Lagrange multipliers proves mathematically that SVM predictions depend only on support vectors, not the entire training set—this sparsity is SVM's secret weapon.