Neural Scaling Laws: The Power Law Relationship
You'll learn: How model loss decreases predictably according to power laws when you scale parameters, data, or compute.
The Core Discovery
When researchers scaled up language models, they discovered something remarkable: performance improvements follow power laws rather than exponential or linear patterns. Specifically, test loss L decreases as a power of three key factors:
- Model size (N parameters):
L ∝ N^(-α) - Dataset size (D tokens):
L ∝ D^(-β) - Compute (C FLOPs):
L ∝ C^(-γ)
Where α, β, and γ are negative exponents (typically around -0.05 to -0.1).
What This Means in Practice
Think of it like diminishing returns with a specific mathematical form. If you double your model from 1B to 2B parameters, you don't get half the loss—you get a reduction proportional to 2^(-α). For α = -0.076, that's about a 5% improvement. To halve the loss, you'd need roughly 100× more parameters.
This relationship holds across many orders of magnitude, making it predictive: you can estimate how much a 100B parameter model will improve over a 10B model without training it.
The Three Axes
Each scaling dimension has independent effects:
- More parameters = more model capacity
- More data = better learning signal
- More compute = ability to train larger models longer
Critically, these are not equal: compute can be allocated to either more parameters or more training steps, and the optimal allocation follows calculable rules.
Key Takeaway: Neural scaling laws reveal that model loss follows predictable power law relationships with parameters, data, and compute—allowing us to forecast performance and optimize resource allocation before expensive training runs.