The Output Projection Matrix
What you'll learn: How the Transformer combines outputs from multiple attention heads back into a single, unified representation using a learned projection matrix.
The Recombination Challenge
After computing attention across multiple heads in parallel, you have multiple separate outputs—one per head. If you have 8 heads, each producing a 64-dimensional output, you now have 8 different 64-dimensional vectors for each token position. But your model expects a single vector of the original model dimension (e.g., 512-d) for downstream processing.
How do you merge these perspectives back together?
What Is W_O?
The output projection matrix (commonly denoted W_O) is a learned weight matrix that combines the concatenated outputs from all attention heads back into the model dimension.
Here's the process:
- Concatenate all head outputs along the feature dimension
- 8 heads × 64 dims = 512-dimensional concatenated vector
- Multiply by W_O to project back to model dimension
- W_O shape:
[model_dim, model_dim](e.g., 512×512)
- W_O shape:
- Output a single vector per token in the original model dimension
Think of it like a committee voting: each head (expert) gives their opinion, you line up all opinions side-by-side, then W_O weighs and blends them into one final decision.
Why Is This Important?
W_O is learned during training, meaning the model discovers how to best combine different attention perspectives. Some heads might contribute more to certain patterns, and W_O learns those optimal weightings.
Key Takeaway: The output projection matrix W_O transforms concatenated multi-head attention outputs back to the model dimension, learning how to optimally blend different attention perspectives into a unified representation.