The Law of Large Numbers
You'll learn: How increasing your sample size makes your average measurements more reliable and closer to the true underlying value.
The Core Idea
Imagine flipping a coin. You know theoretically that a fair coin has a 50% chance of landing heads. But if you flip it just 4 times, you might get 3 heads (75%) or even 4 heads (100%). Does that mean the coin is unfair? Not at all!
The Law of Large Numbers tells us that as you increase the number of coin flips—say to 100, 1,000, or 10,000 flips—your observed percentage of heads will get closer and closer to the true 50%. The sample mean (what you actually observe) converges to the expected value (the theoretical truth) as your sample size grows.
Why This Matters
In machine learning, we constantly work with samples of data to estimate true population parameters:
- Training a model on a dataset to learn patterns in real-world data
- Estimating average user behavior from a subset of users
- Measuring model performance on test data
The Law of Large Numbers assures us that with enough data points, our sample statistics become reliable estimates of the true underlying values. This is why "more data" is often the answer to improving model accuracy—larger samples give us better estimates of expectations like means, error rates, and probabilities.
A Simple Analogy
Think of it like tasting soup: one spoonful might be too salty by chance, but if you mix well and taste many spoonfuls, you'll get an accurate sense of how salty the whole pot really is.
Key Takeaway: As your sample size increases, the sample mean becomes increasingly reliable and converges to the true expected value—making large datasets powerful for learning accurate patterns.