Course contentsShow
← Machine Learning and Deep Learning
Lesson 1074 of 3,53824. The Transformer ArchitectureFree lesson

Head Dimension and Model Dimension Relationship

Why d_k = d_model / num_heads is standard and how this affects capacity and computational efficiency.

Head Dimension and Model Dimension Relationship

What you'll learn: Why splitting the model dimension evenly across attention heads is standard practice and how it affects model capacity and efficiency.

The Standard Formula

In Multi-Head Attention, there's a fundamental relationship between three dimensions:

d_k = d_model / num_heads

Where:

  • d_model is your overall embedding dimension (e.g., 512)
  • num_heads is the number of attention heads (e.g., 8)
  • d_k is the dimension per head (e.g., 64)

Why This Design?

Capacity preservation: If each head had dimension d_model instead of d_model / num_heads, you'd multiply your parameters by num_heads. That's wasteful and harder to train.

Computational efficiency: By splitting rather than duplicating, the total computation remains roughly constant. Computing 8 heads of dimension 64 costs about the same as 1 head of dimension 512, but gives you 8 different learned representations.

Think of it like teamwork: Instead of having 8 experts each analyzing the entire 512-dimensional space (expensive and redundant), you have 8 specialists each focusing on a 64-dimensional subspace. Together, they still cover the full capacity.

The Trade-off

Each individual head has less representational power (d_k is smaller), but having multiple diverse perspectives compensates for this. The final concatenation and output projection recombine these specialized views into the full d_model dimension.

Key Takeaway: Setting d_k = d_model / num_heads maintains total model capacity while enabling parallel, specialized attention patterns—maximizing efficiency without sacrificing expressiveness.