Padding Masks for Variable-Length Sequences
What you'll learn: How to prevent padding tokens from interfering with attention computations in sequences of different lengths.
The Problem with Padding
When you batch sequences together, they need the same length. Shorter sequences get padded with special tokens (often zeros or a <PAD> token). But here's the issue: attention shouldn't care about these fake padding tokens. If padding contributes to attention scores, your model will learn from meaningless data.
What Is a Padding Mask?
A padding mask is a binary matrix (same shape as your attention score matrix) that marks which positions are real tokens (1 or True) and which are padding (0 or False). Before applying softmax to attention scores, you set the scores corresponding to padding positions to negative infinity (-inf).
How It Works
Think of it like a guest list at a party: you only want to pay attention to people who actually showed up, not empty chairs.
- Create a mask:
[True, True, True, False, False]for a sequence with 3 real tokens and 2 padding tokens - In the attention score matrix, set padding positions to
-inf - When softmax is applied,
exp(-inf) = 0, so padding gets zero attention weight - Result: padding tokens contribute nothing to the weighted sum of values
# Conceptual example (building on your attention knowledge)
scores = torch.matmul(Q, K.transpose(-2, -1)) / sqrt(d_k)
# Apply padding mask before softmax
scores = scores.masked_fill(padding_mask == 0, float('-inf'))
attention_weights = F.softmax(scores, dim=-1)
Key Takeaway: Padding masks ensure that meaningless padding tokens receive zero attention weight by setting their scores to negative infinity before softmax, preventing them from corrupting your model's learned representations.