Parameter Count in Multi-Head Attention
What You'll Learn
You'll learn how to calculate the total number of trainable parameters in a multi-head attention mechanism by counting the weights in projection matrices.
Understanding the Parameter Sources
Multi-head attention uses linear projections to transform your inputs into queries (Q), keys (K), and values (V), then combines the results. Each of these transformations requires a weight matrix with learnable parameters.
The Four Key Projection Matrices
For a model with embedding dimension d_model and h attention heads:
- Query projection: Transforms input to queries →
d_model × d_modelparameters - Key projection: Transforms input to keys →
d_model × d_modelparameters - Value projection: Transforms input to values →
d_model × d_modelparameters - Output projection: Combines multi-head results →
d_model × d_modelparameters
Notice that even though you split attention across h heads, the total dimension stays d_model. Each head works with d_model/h dimensions, but all heads together still process the full d_model dimensions.
The Total Count
Total parameters = 4 × (d_model × d_model) = 4d_model²
For example, if d_model = 512:
- Q, K, V projections: 3 × (512 × 512) = 786,432 parameters
- Output projection: 512 × 512 = 262,144 parameters
- Total: 1,048,576 parameters
Why Not Dependent on Number of Heads?
Think of it like slicing a pizza: whether you cut it into 4 or 8 slices, you still have the same amount of pizza. Similarly, splitting d_model across more heads doesn't change the total projection dimensions—just how the work is divided.
Key Takeaway: Multi-head attention uses exactly 4d_model² parameters regardless of the number of heads: three projection matrices for Q, K, and V, plus one output projection to recombine the heads.