Dataset Size and Statistical Power
You'll learn why your evaluation dataset must be large enough to reliably detect when one model truly outperforms another.
The Problem of Small Datasets
Imagine flipping a coin twice and getting two heads. Can you conclude it's unfair? Of course not—you need more flips to be confident. The same logic applies to LLM evaluation. If you test two models on just 10 examples, a model that gets 8/10 correct might just be lucky compared to one that gets 7/10.
Statistical power is your ability to detect a real performance difference when it exists. Too few examples means:
- Random variation drowns out genuine differences
- You can't distinguish between a truly better model and lucky noise
- You waste resources deploying an inferior model (or vice versa)
How Large Is Large Enough?
The minimum dataset size depends on:
- Effect size: How big is the performance gap you need to detect? Spotting a 20% accuracy difference needs fewer examples than detecting 2%
- Metric variance: Binary tasks (correct/incorrect) need fewer examples than subjective 1-5 ratings with human disagreement
- Confidence level: Higher confidence (e.g., 95% vs 80%) requires more data
- Base rate: If your task succeeds 95% of the time, you need many examples to see rare failures
A rough guideline: 100-300 examples minimum for most tasks, but 1000+ if detecting small improvements or measuring rare failure modes.
Practical Impact
With 50 examples, a model scoring 82% vs 78% might not be statistically different. With 500 examples, that same gap becomes meaningful and actionable.
Key Takeaway: Choose dataset sizes that give you the statistical power to confidently detect the minimum performance difference that matters for your application—don't let randomness drive your decisions.