Course contentsShow
← Machine Learning and Deep Learning
Lesson 1072 of 3,53824. The Transformer ArchitectureFree lesson

The Output Projection Matrix

Understanding W_O: how concatenated head outputs are projected back to the model dimension.

The Output Projection Matrix

What you'll learn: How the Transformer combines outputs from multiple attention heads back into a single, unified representation using a learned projection matrix.

The Recombination Challenge

After computing attention across multiple heads in parallel, you have multiple separate outputs—one per head. If you have 8 heads, each producing a 64-dimensional output, you now have 8 different 64-dimensional vectors for each token position. But your model expects a single vector of the original model dimension (e.g., 512-d) for downstream processing.

How do you merge these perspectives back together?

What Is W_O?

The output projection matrix (commonly denoted W_O) is a learned weight matrix that combines the concatenated outputs from all attention heads back into the model dimension.

Here's the process:

  1. Concatenate all head outputs along the feature dimension
    • 8 heads × 64 dims = 512-dimensional concatenated vector
  2. Multiply by W_O to project back to model dimension
    • W_O shape: [model_dim, model_dim] (e.g., 512×512)
  3. Output a single vector per token in the original model dimension

Think of it like a committee voting: each head (expert) gives their opinion, you line up all opinions side-by-side, then W_O weighs and blends them into one final decision.

Why Is This Important?

W_O is learned during training, meaning the model discovers how to best combine different attention perspectives. Some heads might contribute more to certain patterns, and W_O learns those optimal weightings.

Key Takeaway: The output projection matrix W_O transforms concatenated multi-head attention outputs back to the model dimension, learning how to optimally blend different attention perspectives into a unified representation.