Course contentsShow
← Machine Learning and Deep Learning
Lesson 1621 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Parameter Count vs Performance

How increasing model parameters (N) affects loss and capabilities, with diminishing but predictable returns.

Parameter Count vs Performance

What you'll learn: How scaling up the number of parameters in a neural network improves performance predictably, but with diminishing returns.

The Core Relationship

When you increase the number of parameters (N) in a language model—by making it wider, deeper, or both—the model's performance improves consistently. This isn't random: larger models achieve lower loss on held-out data and unlock new capabilities that smaller models simply cannot exhibit.

The relationship follows a power law: as you double the parameter count, you get a predictable reduction in loss, but the gains become smaller each time. Going from 100M to 1B parameters yields a bigger improvement than going from 10B to 100B, even though both represent 10× increases.

Why This Matters

This predictability is revolutionary. It means you can forecast how well a 100B-parameter model will perform before training it, based on smaller experiments. Engineers use this to decide whether the computational cost justifies the gain.

However, diminishing returns mean you can't just throw parameters at every problem. A 10× parameter increase might reduce loss by 20%, but requires 10× more compute and memory. The sweet spot depends on your budget, latency requirements, and task.

The Performance Curve

Think of it like climbing a mountain: early steps (small models) give you altitude quickly. As you climb higher (larger models), each step forward yields less vertical gain. You're still improving, but the efficiency drops.

Key Takeaway: Increasing model parameters N consistently improves performance following a power law, with predictable but diminishing returns—making parameter count a reliable but expensive lever for capability gains.