Law of Large Numbers
What you'll learn: How averaging more and more samples gets you closer and closer to the true expected value.
The Core Idea
Imagine flipping a fair coin. You know the true probability of heads is 0.5, but if you flip it just 3 times, you might get 2 heads (67%) or even 3 heads (100%). That's far from 50%!
The Law of Large Numbers tells us something reassuring: as you flip more coins—10, 100, 1000 times—the average result (proportion of heads) will get closer and closer to the true expected value of 0.5.
In formal terms: as your sample size increases, the sample average converges to the expected value (mean) of the distribution.
Why This Matters in ML
When training models, we:
- Estimate means, variances, and other statistics from samples of data
- Assume these sample estimates represent the true population parameters
The Law of Large Numbers guarantees that with enough data, our sample statistics become reliable. With only 10 training examples, your estimated mean might be way off. With 10,000 examples, it's much more trustworthy.
A Simple Example
Rolling a fair six-sided die has an expected value of 3.5 (the mean of 1,2,3,4,5,6).
- Roll 5 times: average might be 2.8
- Roll 50 times: average might be 3.4
- Roll 5000 times: average will be very close to 3.5
The more samples you collect, the less randomness affects your average.
Key Takeaway: The Law of Large Numbers ensures that larger samples give you more accurate estimates of true population parameters—a fundamental reason why "more data" often improves machine learning models.