Course contentsShow
← Machine Learning and Deep Learning
Lesson 1629 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Inference Cost Scaling

How model size affects inference latency and memory, and the tradeoff between capability and deployability.

Inference Cost Scaling

You'll learn how larger models trade capability for higher deployment costs at inference time.

The Core Problem

When you deploy a large language model for real-world use, every prediction (inference) costs compute resources. As model size grows—more parameters, layers, and hidden dimensions—two critical bottlenecks emerge:

  1. Memory requirements: A 70B parameter model needs ~140GB of memory just to store weights (in float16), while a 7B model needs only ~14GB. This determines what hardware you can even run on.

  2. Latency per token: Larger models perform more matrix multiplications per forward pass. A 70B model might take 10x longer to generate each token compared to a 7B model on the same hardware.

The Capability vs Deployability Tradeoff

Think of it like choosing between a massive diesel truck and a compact car. The truck (large model) can haul more cargo (handle harder tasks, emergent abilities), but it guzzles fuel (compute) and needs special parking (expensive GPUs). The compact car (smaller model) is cheaper to run and fits anywhere, but has limited capacity.

In practice:

  • A 1B model might run on a smartphone
  • A 7B model fits on consumer GPUs
  • A 70B model requires multi-GPU servers or batched cloud inference

Businesses must balance: "Do we need the extra capability enough to justify 10x the infrastructure cost per user query?"

Key Takeaway: Model size directly multiplies inference memory and latency costs—scaling laws tell you what performance you gain, but deployment reality determines what you can actually afford to serve.