Masked Self-Attention in Decoder
What you'll learn: How causal masking prevents the decoder from "cheating" by looking ahead at future tokens during text generation.
The Core Problem
When a Transformer decoder generates text word-by-word (autoregressively), it must predict each token based only on previous tokens—never future ones. Without masking, self-attention would let the decoder peek at all positions, including words it hasn't generated yet. That's cheating during training!
How Causal Masking Works
Masked self-attention applies the mask matrix you learned earlier (upper triangular masking) to the attention scores before the softmax operation. This sets future positions to negative infinity, making their attention weights effectively zero after softmax.
The Mechanism Step-by-Step
- Compute attention scores between all token pairs
- Apply the causal mask: set scores for future positions to
-∞ - Apply softmax (future positions get weight ≈ 0)
- Compute weighted sum of values (future tokens contribute nothing)
Why This Matters
During training, you feed the entire target sequence at once for efficiency. Without masking, the model would see the answer it's supposed to predict! Masking ensures that when predicting position 3, the decoder only attends to positions 0, 1, and 2—mimicking the actual generation process.
Analogy
Think of writing a sentence while blindfolded to everything you haven't written yet. You can review what you've already written (attend to past tokens), but you can't peek ahead at the rest of the page (future tokens are masked).
Key Takeaway: Masked self-attention in the decoder enforces causality by preventing attention to future positions, ensuring the model learns to generate sequences autoregressively without access to information it shouldn't have during prediction.