Course contentsShow
← Machine Learning and Deep Learning
Lesson 84 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Bias and Variance of Estimators

Analyzing estimator quality through bias, variance, and the bias-variance tradeoff.

Bias and Variance of Estimators

What you'll learn: How to judge whether an estimator is reliable by understanding its bias and variance—the two key measures of estimator quality.

The Core Idea

When you use sample data to estimate a population parameter (like the mean), your estimator is a random variable that varies from sample to sample. Two critical properties determine how good your estimator is:

Bias measures systematic error—how far off your estimator is on average from the true value. An estimator is unbiased if its expected value equals the true parameter. Think of it like a dartboard: if your darts cluster around the bullseye on average, you're unbiased.

Variance measures consistency—how much your estimates fluctuate between different samples. Low variance means your estimates are tightly clustered; high variance means they're scattered. Back to darts: tight grouping means low variance, even if you're off-center.

The Bias-Variance Tradeoff

Here's the catch: reducing bias often increases variance, and vice versa.

  • Low bias, high variance: Your estimates are correct on average but wildly inconsistent (darts scattered around the bullseye)
  • High bias, low variance: Your estimates are consistently wrong in the same direction (darts tightly grouped, but far from center)
  • The sweet spot: Minimize total error, which combines both bias² and variance

For example, the sample mean is an unbiased estimator of the population mean (bias = 0), but has variance that depends on sample size. A constant estimator has zero variance but maximum bias.

Why This Matters

Understanding this tradeoff helps you choose better estimators and informs critical machine learning concepts like model complexity and overfitting.

Key Takeaway: An estimator's quality depends on both bias (systematic error) and variance (inconsistency)—good estimators balance both to minimize total error.