Course contentsShow
← Machine Learning and Deep Learning
Lesson 1627 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Layer Count, Hidden Dimension, and Heads

How parameter count translates to architectural choices: number of layers, embedding size, and attention heads.

Layer Count, Hidden Dimension, and Heads

What you'll learn: How the total parameter count of a transformer model breaks down into three key architectural choices: number of layers, hidden dimension size, and number of attention heads.

The Parameter Budget Allocation

When you have a target parameter count (say, 7B parameters), you face a critical design question: how do you distribute those parameters across your architecture? The three main "knobs" are:

  • Layer count (depth): How many transformer blocks to stack
  • Hidden dimension (width): The size of embeddings and feedforward networks
  • Attention heads: How many parallel attention mechanisms per layer

The Interplay

Think of your parameter budget like a fixed amount of money. You can buy a tall, narrow building (many layers, smaller hidden size) or a short, wide building (fewer layers, larger hidden size).

Parameters scale roughly as:

  • Linear with layer count: 2× layers ≈ 2× parameters
  • Quadratic with hidden dimension: 2× hidden size ≈ 4× parameters (due to matrix multiplications)
  • Minimal with head count (heads just partition the hidden dimension)

For example, a 7B model might use: 32 layers × 4096 hidden dimension × 32 heads, while a 13B model might use: 40 layers × 5120 hidden dimension × 40 heads.

Design Constraints

Head count typically divides evenly into hidden dimension (e.g., 4096 dim ÷ 32 heads = 128 dim per head). More heads allow richer attention patterns but don't significantly increase parameters. The real trade-off is depth vs width.

Key Takeaway: Your total parameter count determines your architectural choices—modern LLMs balance many layers (20-80+) with large hidden dimensions (2048-8192+) and proportional head counts, with the hidden dimension being the most "expensive" dimension to scale.