Dimension Splitting vs. Independent Projections
What you'll learn: You'll understand the two ways to implement multi-head attention's linear projections—splitting one large matrix or using separate smaller ones.
The Two Approaches
When implementing multi-head attention, we need to create Query, Key, and Value projections for each attention head. There are two mathematically equivalent ways to do this:
Approach 1: Dimension Splitting
Project the entire d_model dimension with one large weight matrix, then split the result into h heads. Each head gets d_k = d_model / h dimensions.
Analogy: Imagine baking one large pizza, then cutting it into 8 slices. Each person gets their slice from the same pizza.
# One large projection
W_Q = nn.Linear(d_model, d_model) # 512 → 512
Q = W_Q(x) # Shape: (batch, seq_len, 512)
# Then split into 8 heads of 64 dimensions each
Q = Q.view(batch, seq_len, 8, 64)
Approach 2: Independent Projections
Create separate smaller projection matrices for each head from the start. Each head has its own W_Q, W_K, W_V projecting d_model → d_k.
Analogy: Each person orders their own personal pizza with different toppings.
# 8 separate smaller projections
heads = [nn.Linear(d_model, d_k) for _ in range(8)] # 8x (512 → 64)
Why Split Matters
In practice, dimension splitting is standard because:
- Single matrix multiplication is more efficient than 8 separate ones
- Easier to parallelize on GPUs
- Same expressiveness with better performance
Key Takeaway: Dimension splitting uses one large projection matrix then reshapes the output, while independent projections use separate matrices per head—both are equivalent mathematically, but splitting is computationally superior.