Course contentsShow
← Machine Learning and Deep Learning
Lesson 1620 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Neural Scaling Laws: The Power Law Relationship

Understanding the power law form of loss as a function of parameters, dataset size, and training compute.

Neural Scaling Laws: The Power Law Relationship

You'll learn: How model loss decreases predictably according to power laws when you scale parameters, data, or compute.

The Core Discovery

When researchers scaled up language models, they discovered something remarkable: performance improvements follow power laws rather than exponential or linear patterns. Specifically, test loss L decreases as a power of three key factors:

  • Model size (N parameters): L ∝ N^(-α)
  • Dataset size (D tokens): L ∝ D^(-β)
  • Compute (C FLOPs): L ∝ C^(-γ)

Where α, β, and γ are negative exponents (typically around -0.05 to -0.1).

What This Means in Practice

Think of it like diminishing returns with a specific mathematical form. If you double your model from 1B to 2B parameters, you don't get half the loss—you get a reduction proportional to 2^(-α). For α = -0.076, that's about a 5% improvement. To halve the loss, you'd need roughly 100× more parameters.

This relationship holds across many orders of magnitude, making it predictive: you can estimate how much a 100B parameter model will improve over a 10B model without training it.

The Three Axes

Each scaling dimension has independent effects:

  • More parameters = more model capacity
  • More data = better learning signal
  • More compute = ability to train larger models longer

Critically, these are not equal: compute can be allocated to either more parameters or more training steps, and the optimal allocation follows calculable rules.

Key Takeaway: Neural scaling laws reveal that model loss follows predictable power law relationships with parameters, data, and compute—allowing us to forecast performance and optimize resource allocation before expensive training runs.