The Bias-Variance Tradeoff
You'll learn: How model complexity influences the two main sources of prediction error—bias and variance—and why balancing them is crucial for good generalization.
The Core Idea
Every machine learning model makes prediction errors. The bias-variance tradeoff describes how these errors decompose into two competing sources:
-
Bias measures how far off your model's average predictions are from the true values. High bias means your model is too simple—it systematically misses patterns in the data (underfitting).
-
Variance measures how much your model's predictions change when trained on different datasets. High variance means your model is too sensitive—it memorizes noise and specific quirks of the training data (overfitting).
The Tradeoff
As you increase model complexity (more features, deeper trees, more parameters), bias typically decreases—the model can capture more intricate patterns. However, variance increases—the model becomes hypersensitive to training data fluctuations.
Conversely, simpler models have higher bias but lower variance—they make consistent predictions but miss nuances.
An Analogy
Think of archery:
- High bias, low variance: Your arrows cluster tightly but consistently left of the bullseye (systematic error, but consistent).
- Low bias, high variance: Your arrows scatter all around the bullseye (no systematic error, but wildly inconsistent).
- Goal: Arrows tightly clustered at the bullseye (low bias AND low variance).
Finding the Sweet Spot
The total error on new data combines bias, variance, and irreducible noise. Your goal is to find the model complexity that minimizes total error—not too simple, not too complex. This is why we use test data to evaluate generalization and techniques like cross-validation to tune complexity.
Key Takeaway: The bias-variance tradeoff explains why neither the simplest nor most complex model is best—optimal performance requires balancing underfitting and overfitting.