Dataset Size Scaling
You'll learn how increasing the number of training tokens (D) consistently improves model performance according to predictable scaling laws.
The Core Relationship
Just as adding more parameters to your model improves performance following a power law (as we saw in the previous lesson), increasing the amount of training data follows a similar pattern. When researchers train language models on more tokens—whether 100 billion, 500 billion, or several trillion—performance improves predictably.
The relationship looks like:
Loss ∝ D^(-α)
where D is the number of training tokens and α is a positive exponent (typically around 0.095 for language models). This means doubling your dataset size provides a consistent, measurable improvement in loss, which translates to better real-world performance.
Why More Data Always Helps
Think of training data like examples in a textbook. If you've only seen 10 math problems, you might struggle with variations. But after working through 1,000 problems, you've encountered more edge cases, patterns, and problem types. Language models behave similarly—more diverse training examples help them generalize better.
Unlike traditional machine learning where you can "saturate" with enough data, modern large language models haven't shown signs of this ceiling yet. Even at trillions of tokens, the power law continues to hold.
The Practical Implication
This scaling behavior means that data is as important as parameters. A 10B parameter model trained on massive data can outperform a 100B parameter model trained on limited data. The interplay between model size (N) and dataset size (D) is what the complete scaling laws capture—you need to scale both together for optimal results.
Key Takeaway: Model performance scales predictably with dataset size following a power law (Loss ∝ D^(-α)), meaning more training tokens consistently yield better models without apparent saturation limits.