Course contentsShow
← Machine Learning and Deep Learning
Lesson 222 of 3,5383. Linear RegressionFree lesson

Bias-Variance Tradeoff in Regression

Decomposing prediction error into bias, variance, and irreducible error components.

Bias-Variance Tradeoff in Regression

What You'll Learn

You'll learn how to decompose a model's prediction error into three distinct components: bias, variance, and irreducible error.

Understanding the Error Breakdown

When your regression model makes predictions, the total error can be mathematically separated into three parts:

1. Bias Error – How far off your model's average prediction is from the true value. High bias means your model is too simple and consistently misses the mark (underfitting). Think of a archer who always shoots left of the bullseye—their shots are systematically wrong.

2. Variance Error – How much your model's predictions scatter when trained on different datasets. High variance means your model is too sensitive to training data fluctuations (overfitting). Like an archer whose shots land randomly all over the target—no consistency.

3. Irreducible Error – The noise inherent in the data itself that no model can eliminate. This is the fundamental randomness in the real world.

The Mathematical Relationship

The expected prediction error at any point equals:

Total Error = Bias² + Variance + Irreducible Error

The Tradeoff

Here's the crucial insight: as you make your model more complex (adding polynomial features, for example), bias decreases but variance increases. As you simplify it, the opposite happens. You cannot minimize both simultaneously—you must find the sweet spot.

A straight line through curved data has high bias (too simple) but low variance (stable predictions). A 20th-degree polynomial has low bias (fits training data perfectly) but high variance (wild predictions on new data).

Key Takeaway: Understanding that prediction error decomposes into bias, variance, and irreducible error helps you diagnose whether your model needs more complexity (high bias) or more regularization (high variance).