Course contentsShow
← Machine Learning and Deep Learning
Lesson 1624 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

FLOPs Budget and Training Cost

Estimating computational cost in floating-point operations and how it scales with model and data size.

FLOPs Budget and Training Cost

What You'll Learn

You'll learn how to estimate the computational cost of training large language models in floating-point operations (FLOPs) and understand how this cost scales with model architecture and data.

Understanding FLOPs

A FLOP (floating-point operation) is a single arithmetic operation like addition or multiplication on decimal numbers. When training neural networks, we measure total computation in FLOPs—think of it as "how many calculator button presses" your model requires.

For large language models, the dominant cost comes from matrix multiplications in attention layers and feedforward networks. The training cost formula is surprisingly clean:

FLOPs ≈ 6 × N × D

Where:

  • N = number of model parameters
  • D = number of training tokens
  • The factor of 6 accounts for forward pass (1×), backward pass (2×), and other overhead

Why This Matters

If you train a 7-billion parameter model on 1 trillion tokens:

  • FLOPs ≈ 6 × 7×10⁹ × 1×10¹² = 4.2×10²² FLOPs
  • On an NVIDIA A100 GPU (~300 TFLOPS), that's roughly 1.6 million GPU-hours

This explains why the Chinchilla result was revolutionary: it showed that many models were undertrained (too few tokens for their parameter count), wasting compute by scaling parameters instead of data.

The Trade-off

Your FLOPs budget is fixed by money and time. You must choose:

  • Bigger model, less data? (GPT-3 style)
  • Smaller model, more data? (Chinchilla style)

The compute-optimal frontier tells you the best balance.

Key Takeaway: Training cost in FLOPs scales linearly with both model parameters and training tokens (≈6ND formula), making it the fundamental constraint in deciding model size and dataset size for large language models.