Hinge Loss Interpretation
What you'll learn: How soft-margin SVM can be reframed as minimizing a loss function (hinge loss) plus regularization, just like other machine learning models.
Connecting SVM to Loss-Based Learning
You've seen soft-margin SVM as an optimization problem with slack variables and the C hyperparameter. Now here's a powerful insight: this is actually the same as minimizing a specific loss function called hinge loss, plus L2 regularization.
What is Hinge Loss?
The hinge loss for a single training example is:
L_hinge(y, f(x)) = max(0, 1 - y·f(x))
where:
yis the true label (+1 or -1)f(x)is your model's raw prediction (not thresholded)
How It Works
- If your prediction is correct and beyond the margin (
y·f(x) ≥ 1), loss = 0 - Otherwise, you pay a penalty proportional to how far inside the margin you are
- It "hinges" at the margin boundary—hence the name
The Unified View
Soft-margin SVM minimizes:
(1/2)||w||² + C·Σ(slack_i)
This is equivalent to:
Σ max(0, 1 - y_i·f(x_i)) + λ||w||²
where λ = 1/(2C). The first term is total hinge loss; the second is L2 regularization (just like Ridge regression!).
Why This Matters
This reinterpretation shows SVM fits into the broader framework of regularized empirical risk minimization: minimize training loss + keep model simple. Just like logistic regression uses cross-entropy loss, SVM uses hinge loss.
Key Takeaway: Soft-margin SVM minimizes hinge loss plus L2 regularization, revealing it as a standard loss-based learning model that penalizes margin violations while controlling model complexity.