Course contentsShow
← Machine Learning and Deep Learning
Lesson 1077 of 3,53824. The Transformer ArchitectureFree lesson

Masked Multi-Head Attention

Extending multi-head attention with causal masking for autoregressive decoding in transformers.

Masked Multi-Head Attention

What you'll learn: How to extend multi-head attention with causal masking to prevent future information leakage in autoregressive transformers.

The Core Problem

You've already learned about multi-head attention (lesson 1068-1076) and causal masking (lesson 1060-1061). Now we combine them! In autoregressive models—like GPT or the decoder side of a transformer—we need to ensure that when predicting the next token, the model can only "see" previous tokens, never future ones.

How It Works

Masked multi-head attention applies the upper triangular mask inside each attention head during the scaled dot-product computation. Here's the flow:

  1. Compute Q, K, V projections for all heads (just like regular multi-head attention)
  2. Calculate attention scores: QK^T / √d_k
  3. Apply the mask by setting future positions to -inf before softmax
  4. Apply softmax to get attention weights (future positions become 0)
  5. Compute weighted sum of values
  6. Concatenate heads and apply output projection

The key difference: the mask prevents position i from attending to positions j > i, enforcing the autoregressive property across all heads simultaneously.

Why It Matters

This mechanism is what allows transformer decoders to generate text one token at a time during inference while still training efficiently in parallel. During training, the entire target sequence is processed at once, but the mask ensures each position only uses valid historical context—exactly matching the sequential generation behavior at inference time.

Key Takeaway: Masked multi-head attention combines the representational power of multiple attention heads with causal masking to enable parallel training of autoregressive models while maintaining the constraint that predictions at each position depend only on previous positions.