Compute-Optimal Training: The Chinchilla Result
What you'll learn: How DeepMind's Chinchilla research revealed that most large language models are dramatically undertrained, and what the optimal balance between model size and training data should be.
The Traditional Approach Was Wrong
For years, the focus in scaling language models was simple: make the model bigger. GPT-3 had 175 billion parameters, and conventional wisdom suggested that bigger models automatically meant better performance. The training dataset size was often treated as a secondary concern.
The Chinchilla paper changed everything.
The Chinchilla Discovery
DeepMind researchers trained a 70B parameter model called Chinchilla and compared it to their earlier 280B parameter Gopher model. Surprisingly, Chinchilla outperformed Gopher despite being 4× smaller. The secret? Chinchilla was trained on far more tokens.
Their key finding: For every doubling of model parameters, you should double the training tokens too.
The compute-optimal ratio is roughly 20 tokens per parameter for the training budget used. For a 10B parameter model, you should train on ~200B tokens. For 70B parameters, ~1.4 trillion tokens.
Why Most Models Are Undertrained
Before Chinchilla, models like GPT-3 (175B parameters, 300B tokens) had a ratio of less than 2 tokens per parameter. They were massively undertrained! The compute budget was spent on making models bigger rather than training them longer on more data.
Think of it like building muscle: you could train with heavier weights for fewer reps (big model, little data) or lighter weights for many more reps (smaller model, more data). Chinchilla showed the second approach is more efficient.
Key Takeaway: Compute-optimal training requires balancing model size with training data — most large models are undertrained, and training smaller models on more tokens often yields better performance per unit of compute.