Masked Multi-Head Attention
What you'll learn: How to extend multi-head attention with causal masking to prevent future information leakage in autoregressive transformers.
The Core Problem
You've already learned about multi-head attention (lesson 1068-1076) and causal masking (lesson 1060-1061). Now we combine them! In autoregressive models—like GPT or the decoder side of a transformer—we need to ensure that when predicting the next token, the model can only "see" previous tokens, never future ones.
How It Works
Masked multi-head attention applies the upper triangular mask inside each attention head during the scaled dot-product computation. Here's the flow:
- Compute Q, K, V projections for all heads (just like regular multi-head attention)
- Calculate attention scores:
QK^T / √d_k - Apply the mask by setting future positions to
-infbefore softmax - Apply softmax to get attention weights (future positions become 0)
- Compute weighted sum of values
- Concatenate heads and apply output projection
The key difference: the mask prevents position i from attending to positions j > i, enforcing the autoregressive property across all heads simultaneously.
Why It Matters
This mechanism is what allows transformer decoders to generate text one token at a time during inference while still training efficiently in parallel. During training, the entire target sequence is processed at once, but the mask ensures each position only uses valid historical context—exactly matching the sequential generation behavior at inference time.
Key Takeaway: Masked multi-head attention combines the representational power of multiple attention heads with causal masking to enable parallel training of autoregressive models while maintaining the constraint that predictions at each position depend only on previous positions.