Course contentsShow
← Machine Learning and Deep Learning
Lesson 73 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Law of Large Numbers

Understand how sample averages converge to expected values as sample size increases.

Law of Large Numbers

What you'll learn: How averaging more and more samples gets you closer and closer to the true expected value.

The Core Idea

Imagine flipping a fair coin. You know the true probability of heads is 0.5, but if you flip it just 3 times, you might get 2 heads (67%) or even 3 heads (100%). That's far from 50%!

The Law of Large Numbers tells us something reassuring: as you flip more coins—10, 100, 1000 times—the average result (proportion of heads) will get closer and closer to the true expected value of 0.5.

In formal terms: as your sample size increases, the sample average converges to the expected value (mean) of the distribution.

Why This Matters in ML

When training models, we:

  • Estimate means, variances, and other statistics from samples of data
  • Assume these sample estimates represent the true population parameters

The Law of Large Numbers guarantees that with enough data, our sample statistics become reliable. With only 10 training examples, your estimated mean might be way off. With 10,000 examples, it's much more trustworthy.

A Simple Example

Rolling a fair six-sided die has an expected value of 3.5 (the mean of 1,2,3,4,5,6).

  • Roll 5 times: average might be 2.8
  • Roll 50 times: average might be 3.4
  • Roll 5000 times: average will be very close to 3.5

The more samples you collect, the less randomness affects your average.

Key Takeaway: The Law of Large Numbers ensures that larger samples give you more accurate estimates of true population parameters—a fundamental reason why "more data" often improves machine learning models.