The Mask Matrix: Upper Triangular Masking
What you'll learn: How to construct and apply attention masks using -∞ values before softmax to prevent positions from attending to future tokens.
The Problem: Peeking Into the Future
In autoregressive models (like language models that generate text one token at a time), each position should only attend to tokens that came before it, never to future tokens. Without this constraint, the model would "cheat" during training by looking ahead at the answer.
The Solution: Masking with -∞
Before applying softmax to attention scores, we add a mask matrix that sets certain positions to -∞. When softmax processes -∞, it converts these values to 0, effectively preventing attention flow.
The mask is an upper triangular matrix of -∞ values (above the diagonal):
Position: 0 1 2 3
0 [ 0, -∞, -∞, -∞ ]
1 [ 0, 0, -∞, -∞ ]
2 [ 0, 0, 0, -∞ ]
3 [ 0, 0, 0, 0 ]
Position 0 can only attend to itself. Position 1 can attend to positions 0 and 1. Position 3 can attend to all previous positions.
How It Works
- Compute attention scores (dot products scaled by √d_k)
- Add the mask matrix element-wise to scores
- Apply softmax — the -∞ values become 0 weights
- Compute weighted sum of values
This ensures that when predicting token at position i, the model cannot access information from positions i+1 onwards.
Key Takeaway: Upper triangular masking with -∞ values enforces causal attention by zeroing out attention weights to future positions, essential for autoregressive generation during training.