The C Hyperparameter
What you'll learn: How the C hyperparameter in soft-margin SVMs controls the balance between maximizing the margin and minimizing training errors.
The Core Idea
In the previous lesson, you learned that soft-margin SVMs introduce slack variables to allow some misclassifications or margin violations. But how do we decide how much error to tolerate?
The C hyperparameter is your control knob for this trade-off. It determines how heavily the SVM penalizes training errors (violations of the margin or misclassifications).
Understanding C
Think of C as the "strictness level" of your classifier:
-
Large C (e.g., C = 1000): Very strict. The model heavily penalizes any training errors, trying to classify every training point correctly. This leads to a narrower margin but fewer misclassifications on training data. Risk: overfitting.
-
Small C (e.g., C = 0.01): More lenient. The model tolerates more training errors in favor of a wider margin. This prioritizes better generalization over perfect training accuracy. Risk: underfitting.
The Mathematical Connection
Recall the soft-margin SVM objective includes both margin maximization and slack penalty terms. The C parameter weights the importance of the slack (error) term. High C means "care a lot about minimizing errors," while low C means "care more about maximizing margin."
An Analogy
Imagine you're drawing a path between two groups of people. A high C is like insisting the path stays far from everyone, even if it becomes very narrow. A low C is like drawing a wide, comfortable path, accepting that a few people might be close to or even on the wrong side.
Key Takeaway: The C hyperparameter controls the bias-variance tradeoff in SVMs—higher C reduces bias but increases variance (potential overfitting), while lower C increases bias but reduces variance (better generalization).