Layer Count, Hidden Dimension, and Heads
What you'll learn: How the total parameter count of a transformer model breaks down into three key architectural choices: number of layers, hidden dimension size, and number of attention heads.
The Parameter Budget Allocation
When you have a target parameter count (say, 7B parameters), you face a critical design question: how do you distribute those parameters across your architecture? The three main "knobs" are:
- Layer count (depth): How many transformer blocks to stack
- Hidden dimension (width): The size of embeddings and feedforward networks
- Attention heads: How many parallel attention mechanisms per layer
The Interplay
Think of your parameter budget like a fixed amount of money. You can buy a tall, narrow building (many layers, smaller hidden size) or a short, wide building (fewer layers, larger hidden size).
Parameters scale roughly as:
- Linear with layer count: 2× layers ≈ 2× parameters
- Quadratic with hidden dimension: 2× hidden size ≈ 4× parameters (due to matrix multiplications)
- Minimal with head count (heads just partition the hidden dimension)
For example, a 7B model might use: 32 layers × 4096 hidden dimension × 32 heads, while a 13B model might use: 40 layers × 5120 hidden dimension × 40 heads.
Design Constraints
Head count typically divides evenly into hidden dimension (e.g., 4096 dim ÷ 32 heads = 128 dim per head). More heads allow richer attention patterns but don't significantly increase parameters. The real trade-off is depth vs width.
Key Takeaway: Your total parameter count determines your architectural choices—modern LLMs balance many layers (20-80+) with large hidden dimensions (2048-8192+) and proportional head counts, with the hidden dimension being the most "expensive" dimension to scale.