Course contentsShow
← Machine Learning and Deep Learning
Lesson 1623 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Compute-Optimal Training: The Chinchilla Result

How Chinchilla showed that models are often undertrained, and the optimal ratio of parameters to training tokens.

Compute-Optimal Training: The Chinchilla Result

What you'll learn: How DeepMind's Chinchilla research revealed that most large language models are dramatically undertrained, and what the optimal balance between model size and training data should be.

The Traditional Approach Was Wrong

For years, the focus in scaling language models was simple: make the model bigger. GPT-3 had 175 billion parameters, and conventional wisdom suggested that bigger models automatically meant better performance. The training dataset size was often treated as a secondary concern.

The Chinchilla paper changed everything.

The Chinchilla Discovery

DeepMind researchers trained a 70B parameter model called Chinchilla and compared it to their earlier 280B parameter Gopher model. Surprisingly, Chinchilla outperformed Gopher despite being 4× smaller. The secret? Chinchilla was trained on far more tokens.

Their key finding: For every doubling of model parameters, you should double the training tokens too.

The compute-optimal ratio is roughly 20 tokens per parameter for the training budget used. For a 10B parameter model, you should train on ~200B tokens. For 70B parameters, ~1.4 trillion tokens.

Why Most Models Are Undertrained

Before Chinchilla, models like GPT-3 (175B parameters, 300B tokens) had a ratio of less than 2 tokens per parameter. They were massively undertrained! The compute budget was spent on making models bigger rather than training them longer on more data.

Think of it like building muscle: you could train with heavier weights for fewer reps (big model, little data) or lighter weights for many more reps (smaller model, more data). Chinchilla showed the second approach is more efficient.

Key Takeaway: Compute-optimal training requires balancing model size with training data — most large models are undertrained, and training smaller models on more tokens often yields better performance per unit of compute.