Course contentsShow
← Machine Learning and Deep Learning
Lesson 1070 of 3,53824. The Transformer ArchitectureFree lesson

Dimension Splitting vs. Independent Projections

Comparing two implementation approaches: splitting d_model or using separate smaller projections per head.

Dimension Splitting vs. Independent Projections

What you'll learn: You'll understand the two ways to implement multi-head attention's linear projections—splitting one large matrix or using separate smaller ones.

The Two Approaches

When implementing multi-head attention, we need to create Query, Key, and Value projections for each attention head. There are two mathematically equivalent ways to do this:

Approach 1: Dimension Splitting

Project the entire d_model dimension with one large weight matrix, then split the result into h heads. Each head gets d_k = d_model / h dimensions.

Analogy: Imagine baking one large pizza, then cutting it into 8 slices. Each person gets their slice from the same pizza.

# One large projection
W_Q = nn.Linear(d_model, d_model)  # 512 → 512
Q = W_Q(x)  # Shape: (batch, seq_len, 512)
# Then split into 8 heads of 64 dimensions each
Q = Q.view(batch, seq_len, 8, 64)

Approach 2: Independent Projections

Create separate smaller projection matrices for each head from the start. Each head has its own W_Q, W_K, W_V projecting d_model → d_k.

Analogy: Each person orders their own personal pizza with different toppings.

# 8 separate smaller projections
heads = [nn.Linear(d_model, d_k) for _ in range(8)]  # 8x (512 → 64)

Why Split Matters

In practice, dimension splitting is standard because:

  • Single matrix multiplication is more efficient than 8 separate ones
  • Easier to parallelize on GPUs
  • Same expressiveness with better performance

Key Takeaway: Dimension splitting uses one large projection matrix then reshapes the output, while independent projections use separate matrices per head—both are equivalent mathematically, but splitting is computationally superior.