Regularization in Logistic Regression
What you'll learn: How to apply L1 and L2 regularization to logistic regression models to prevent overfitting and automatically select important features.
Why Regularize Logistic Regression?
Just like linear regression, logistic regression can overfit—especially when you have many features or complex decision boundaries. The model might memorize training data patterns that don't generalize. Regularization adds a penalty to the loss function that discourages overly complex models.
Applying L1 and L2 to Logistic Regression
Recall that logistic regression minimizes cross-entropy loss. With regularization, you add a penalty term:
L2 (Ridge) Regularization:
- Adds λ × (sum of squared weights) to the loss function
- Shrinks all weights toward zero proportionally
- Keeps all features but reduces their influence
- Creates smoother, more stable decision boundaries
L1 (Lasso) Regularization:
- Adds λ × (sum of absolute values of weights) to the loss function
- Drives some weights exactly to zero
- Performs automatic feature selection by eliminating irrelevant features
- Produces sparse models that are easier to interpret
The Feature Selection Advantage
Think of L1 regularization as a strict hiring manager: it zeros out weights for features that don't pull their weight, effectively removing them from the model. L2 is more lenient—it reduces everyone's influence but keeps everyone on the team.
For high-dimensional data with potentially irrelevant features (e.g., text classification with thousands of words), L1 regularization is particularly valuable because it tells you which features actually matter for prediction.
Choosing λ (Lambda)
The regularization strength λ controls the tradeoff: higher λ means more penalty (simpler model), lower λ means less penalty (risk of overfitting). Use cross-validation to find the optimal value.
Key Takeaway: L1 and L2 regularization prevent logistic regression from overfitting by penalizing large weights, with L1 uniquely offering automatic feature selection by driving irrelevant weights to exactly zero.