Head Dimension and Model Dimension Relationship
What you'll learn: Why splitting the model dimension evenly across attention heads is standard practice and how it affects model capacity and efficiency.
The Standard Formula
In Multi-Head Attention, there's a fundamental relationship between three dimensions:
d_k = d_model / num_heads
Where:
d_modelis your overall embedding dimension (e.g., 512)num_headsis the number of attention heads (e.g., 8)d_kis the dimension per head (e.g., 64)
Why This Design?
Capacity preservation: If each head had dimension d_model instead of d_model / num_heads, you'd multiply your parameters by num_heads. That's wasteful and harder to train.
Computational efficiency: By splitting rather than duplicating, the total computation remains roughly constant. Computing 8 heads of dimension 64 costs about the same as 1 head of dimension 512, but gives you 8 different learned representations.
Think of it like teamwork: Instead of having 8 experts each analyzing the entire 512-dimensional space (expensive and redundant), you have 8 specialists each focusing on a 64-dimensional subspace. Together, they still cover the full capacity.
The Trade-off
Each individual head has less representational power (d_k is smaller), but having multiple diverse perspectives compensates for this. The final concatenation and output projection recombine these specialized views into the full d_model dimension.
Key Takeaway: Setting d_k = d_model / num_heads maintains total model capacity while enabling parallel, specialized attention patterns—maximizing efficiency without sacrificing expressiveness.