Course contentsShow
← Data Science
Lesson 631 of 2,14516. Multiple Linear RegressionFree lesson

Cross-Validation for Model Selection

Use k-fold cross-validation to compare models based on out-of-sample predictive performance.

Cross-Validation for Model Selection

You'll learn how to use k-fold cross-validation to choose between competing regression models based on how well they predict new, unseen data.

The Core Idea

When comparing multiple regression models, metrics like AIC, BIC, and adjusted R-squared give you in-sample estimates of performance—they're calculated on the same data you used to fit the model. Cross-validation takes a different approach: it simulates how well each model would perform on truly new data by systematically holding out portions of your dataset, training on the rest, and measuring prediction errors.

How k-Fold Cross-Validation Works

Think of your data like a deck of cards shuffled into k equal piles (commonly k=5 or k=10). Here's the process:

  1. Hold out pile #1 as a "test set"
  2. Train your model on piles #2 through #k
  3. Predict the held-out pile and measure error (like mean squared error)
  4. Repeat, holding out pile #2, then #3, and so on
  5. Average the k error measurements

The model with the lowest average prediction error across all folds typically generalizes best to new data.

Why This Matters for Model Selection

You've already learned several model comparison tools—F-tests for nested models, AIC/BIC penalizing complexity. Cross-validation adds another dimension: actual predictive performance. It's especially valuable when models aren't nested or when you care primarily about prediction rather than inference.

Cross-validation directly answers: "If I collected new data tomorrow, which model would make better predictions?" It naturally penalizes overfitting because overly complex models perform worse on held-out data.

Key Takeaway: Cross-validation estimates out-of-sample prediction error by repeatedly training models on different subsets of your data and testing on the remainder—the model with lowest average error is usually your best choice for prediction tasks.