Multi-Head Attention Architecture
What you'll learn: How multi-head attention splits input across parallel attention heads, processes them independently, and combines the results.
Now that you understand why we need multiple attention heads (to capture different relationship types simultaneously), let's see exactly how the architecture works.
The Split-Process-Concatenate Pipeline
Multi-head attention doesn't simply run attention multiple times on the same inputs. Instead, it follows a structured three-step process:
1. Split via Linear Projections
The input queries (Q), keys (K), and values (V) are each passed through h different learned linear transformations (where h is the number of heads). Each head gets its own set of weight matrices: W^Q_i, W^K_i, W^V_i. This creates h different "views" of the same data, with smaller dimensionality per head.
2. Parallel Attention
Each head independently computes scaled dot-product attention on its projected Q, K, V triplet. All heads run in parallel (no sequential dependency), allowing efficient GPU computation. Each head can learn to focus on different patterns—one might capture positional relationships, another semantic similarity, etc.
3. Concatenate and Project
The outputs from all h heads are concatenated back together into a single matrix, then passed through a final linear transformation (W^O) to produce the multi-head attention output.
The Dimension Split
If your model dimension is 512 and you use 8 heads, each head works with dimension 64 (512 ÷ 8). This keeps the total computation comparable to single-head attention while gaining representational diversity.
Key Takeaway: Multi-head attention splits input through learned projections, runs multiple attention operations in parallel (each capturing different patterns), then concatenates and projects the results—combining diverse perspectives into a rich, unified representation.