Causal Attention Masking
What You'll Learn
You'll learn how to implement causal masking in self-attention mechanisms to ensure autoregressive language models can only attend to past tokens, never future ones.
The Core Problem
In autoregressive language modeling (like GPT), the model predicts the next word based only on previous words. During training, however, we have the complete sentence available. Without caution, the self-attention mechanism would let each token "peek" at all future tokens—essentially giving the model the answers during training!
Causal masking solves this by blocking attention to future positions, enforcing the left-to-right constraint.
How It Works
Think of reading a book with strategic sticky notes: you can look back at any page you've already read, but all future pages are covered. That's causal masking.
In practice, we create a mask matrix applied during attention score computation:
- For position i, the model can attend to positions 0 through i (inclusive)
- Positions i+1 onward are masked out (typically set to negative infinity before softmax)
This creates a lower-triangular attention pattern: each token sees itself and everything before it, but nothing after.
The Masking Matrix
For a sequence of 4 tokens, the causal mask looks like:
[1, 0, 0, 0] ← Token 0 sees only itself
[1, 1, 0, 0] ← Token 1 sees tokens 0,1
[1, 1, 1, 0] ← Token 2 sees tokens 0,1,2
[1, 1, 1, 1] ← Token 3 sees tokens 0,1,2,3
Where 1 = allowed, 0 = masked (blocked).
During attention computation, masked positions get -inf added before softmax, making their attention weights effectively zero.
Key Takeaway: Causal attention masking enforces the autoregressive property during training by preventing tokens from attending to future positions, ensuring the model learns to predict based only on past context—exactly what it must do at inference time.