Course contentsShow
← Machine Learning and Deep Learning
Lesson 221 of 3,5383. Linear RegressionFree lesson

The Problem of Overfitting in Linear Regression

Understanding when and why linear models overfit, and recognizing the signs of high variance in predictions.

The Problem of Overfitting in Linear Regression

What you'll learn: How linear regression models can memorize training data instead of learning true patterns, and how to spot the warning signs.

What Is Overfitting?

Imagine you're studying for a test by memorizing every single practice problem word-for-word, including the typos and irrelevant details. When the actual test arrives with slightly different questions, you panic—you've memorized answers, not understood concepts.

Overfitting in linear regression happens exactly this way: your model learns the training data too well, including its noise and quirks, rather than capturing the true underlying relationship between features and target.

When Linear Models Overfit

Linear regression overfits when you:

  • Use too many features relative to your dataset size
  • Include high-degree polynomial features (like x², x³, x⁴...)
  • Have highly correlated features (multicollinearity)
  • Work with noisy data and a complex feature space

The model becomes overly flexible, twisting itself into complicated shapes to fit every single training point perfectly—even the outliers and random fluctuations.

Recognizing High Variance

The telltale sign of overfitting is high variance: your model's predictions swing wildly with small changes in training data. Practically, you'll see:

  • Near-perfect training performance (very low MSE, R² ≈ 1.0)
  • Poor test/validation performance (much higher MSE, low R²)
  • Large coefficient values that seem unreasonable
  • Unstable predictions on new, unseen data

This massive gap between training and test performance screams "overfitting!" Your model has essentially memorized the training set rather than learned a generalizable pattern.

Key Takeaway: Overfitting occurs when your linear model captures noise instead of signal, resulting in excellent training performance but poor generalization—watch for the train-test performance gap as your primary warning sign.