Course contentsShow
← Computer Science
Lesson 2782 of 2,87262. Machine Learning FundamentalsFree lesson

The Bias-Variance Tradeoff

Understanding how model simplicity (bias) and sensitivity to data (variance) affect predictive performance.

The Bias-Variance Tradeoff

You'll learn: How model complexity influences the two main sources of prediction error—bias and variance—and why balancing them is crucial for good generalization.

The Core Idea

Every machine learning model makes prediction errors. The bias-variance tradeoff describes how these errors decompose into two competing sources:

  • Bias measures how far off your model's average predictions are from the true values. High bias means your model is too simple—it systematically misses patterns in the data (underfitting).

  • Variance measures how much your model's predictions change when trained on different datasets. High variance means your model is too sensitive—it memorizes noise and specific quirks of the training data (overfitting).

The Tradeoff

As you increase model complexity (more features, deeper trees, more parameters), bias typically decreases—the model can capture more intricate patterns. However, variance increases—the model becomes hypersensitive to training data fluctuations.

Conversely, simpler models have higher bias but lower variance—they make consistent predictions but miss nuances.

An Analogy

Think of archery:

  • High bias, low variance: Your arrows cluster tightly but consistently left of the bullseye (systematic error, but consistent).
  • Low bias, high variance: Your arrows scatter all around the bullseye (no systematic error, but wildly inconsistent).
  • Goal: Arrows tightly clustered at the bullseye (low bias AND low variance).

Finding the Sweet Spot

The total error on new data combines bias, variance, and irreducible noise. Your goal is to find the model complexity that minimizes total error—not too simple, not too complex. This is why we use test data to evaluate generalization and techniques like cross-validation to tune complexity.

Key Takeaway: The bias-variance tradeoff explains why neither the simplest nor most complex model is best—optimal performance requires balancing underfitting and overfitting.