Course contentsShow
← Machine Learning and Deep Learning
Lesson 230 of 3,5383. Linear RegressionFree lesson

Choosing the Regularization Parameter

Using cross-validation to select optimal lambda values that balance fit and complexity.

Choosing the Regularization Parameter

What you'll learn: How to select the optimal regularization strength (lambda/alpha) that prevents overfitting without sacrificing too much model performance.

The Goldilocks Problem

You've learned about Ridge, Lasso, and Elastic Net regularization, each controlled by a parameter (often called λ or alpha) that determines how strongly to penalize model complexity. But how do you choose the right value? Too small, and you're back to overfitting. Too large, and your model becomes too simple (underfitting).

Cross-Validation to the Rescue

The solution is cross-validation — a technique you've already seen with cross_val_score. Instead of guessing lambda, you test multiple candidates systematically:

  1. Choose a range of lambda values (e.g., 0.001, 0.01, 0.1, 1, 10, 100)
  2. For each lambda, perform k-fold cross-validation
  3. Calculate average validation error across folds
  4. Select the lambda with the lowest validation error

This process finds the sweet spot: enough regularization to generalize well, but not so much that the model can't capture genuine patterns.

Think of It Like Seasoning

Imagine adjusting salt in a recipe. Too little and it's bland (underfitting), too much and it's inedible (overfitting in reverse). You taste-test different amounts (cross-validation) until you find the perfect balance that most people enjoy (best validation score).

The Practical Approach

In Scikit-learn, GridSearchCV (which you've learned) automates this entire process. You provide candidate lambda values, and it performs cross-validation for each one, returning the optimal choice automatically.

Key Takeaway: Cross-validation lets you objectively select the regularization parameter that balances model complexity and predictive performance, avoiding both overfitting and underfitting.