Course contentsShow
← Machine Learning and Deep Learning
Lesson 1073 of 3,53824. The Transformer ArchitectureFree lesson

Parameter Count in Multi-Head Attention

Calculating the total parameters: Q, K, V projections per head plus the output projection matrix.

Parameter Count in Multi-Head Attention

What You'll Learn

You'll learn how to calculate the total number of trainable parameters in a multi-head attention mechanism by counting the weights in projection matrices.

Understanding the Parameter Sources

Multi-head attention uses linear projections to transform your inputs into queries (Q), keys (K), and values (V), then combines the results. Each of these transformations requires a weight matrix with learnable parameters.

The Four Key Projection Matrices

For a model with embedding dimension d_model and h attention heads:

  1. Query projection: Transforms input to queries → d_model × d_model parameters
  2. Key projection: Transforms input to keys → d_model × d_model parameters
  3. Value projection: Transforms input to values → d_model × d_model parameters
  4. Output projection: Combines multi-head results → d_model × d_model parameters

Notice that even though you split attention across h heads, the total dimension stays d_model. Each head works with d_model/h dimensions, but all heads together still process the full d_model dimensions.

The Total Count

Total parameters = 4 × (d_model × d_model) = 4d_model²

For example, if d_model = 512:

  • Q, K, V projections: 3 × (512 × 512) = 786,432 parameters
  • Output projection: 512 × 512 = 262,144 parameters
  • Total: 1,048,576 parameters

Why Not Dependent on Number of Heads?

Think of it like slicing a pizza: whether you cut it into 4 or 8 slices, you still have the same amount of pizza. Similarly, splitting d_model across more heads doesn't change the total projection dimensions—just how the work is divided.

Key Takeaway: Multi-head attention uses exactly 4d_model² parameters regardless of the number of heads: three projection matrices for Q, K, and V, plus one output projection to recombine the heads.