Computing Attention Scores in Parallel
What you'll learn: How multi-head attention computes all attention scores simultaneously using efficient batched matrix operations.
The Core Idea
In multi-head attention, we don't compute each head's attention scores one at a time — that would be slow! Instead, we leverage batched matrix operations to compute attention for all heads in parallel using a single set of efficient matrix multiplications.
Think of it like a restaurant with multiple chefs (heads) all preparing different dishes simultaneously, rather than one chef cooking each dish sequentially. The kitchen (GPU) has the capacity to handle all chefs at once.
How Parallelization Works
After creating linear projections for queries, keys, and values (as we learned), we reshape these tensors to include an explicit "head" dimension:
- Before reshaping:
(batch_size, seq_len, d_model) - After reshaping:
(batch_size, num_heads, seq_len, d_k)
Where d_k = d_model / num_heads is the dimension per head.
Now, when we compute the scaled dot-product attention score matrix QK^T / √d_k, the operation happens across all heads simultaneously. The batch and head dimensions are treated as independent parallel operations.
The softmax normalization and weighted sum of values (attention_weights @ V) also execute in parallel across all heads at once — no loops required!
Why This Matters
This parallelization is what makes multi-head attention practical. Without it, computing 8 or 16 attention heads would be 8–16× slower. GPUs excel at these batched matrix operations, making the overhead of multiple heads minimal compared to the representational benefits.
Key Takeaway: Multi-head attention computes all heads in parallel by reshaping tensors to include a head dimension, then using batched matrix operations — enabling efficient simultaneous computation across all attention heads without sequential loops.