Course contentsShow
← Machine Learning and Deep Learning
Lesson 1063 of 3,53824. The Transformer ArchitectureFree lesson

Padding Masks for Variable-Length Sequences

Learn to mask padding tokens so they don't contribute to attention computations.

Padding Masks for Variable-Length Sequences

What you'll learn: How to prevent padding tokens from interfering with attention computations in sequences of different lengths.

The Problem with Padding

When you batch sequences together, they need the same length. Shorter sequences get padded with special tokens (often zeros or a <PAD> token). But here's the issue: attention shouldn't care about these fake padding tokens. If padding contributes to attention scores, your model will learn from meaningless data.

What Is a Padding Mask?

A padding mask is a binary matrix (same shape as your attention score matrix) that marks which positions are real tokens (1 or True) and which are padding (0 or False). Before applying softmax to attention scores, you set the scores corresponding to padding positions to negative infinity (-inf).

How It Works

Think of it like a guest list at a party: you only want to pay attention to people who actually showed up, not empty chairs.

  1. Create a mask: [True, True, True, False, False] for a sequence with 3 real tokens and 2 padding tokens
  2. In the attention score matrix, set padding positions to -inf
  3. When softmax is applied, exp(-inf) = 0, so padding gets zero attention weight
  4. Result: padding tokens contribute nothing to the weighted sum of values
# Conceptual example (building on your attention knowledge)
scores = torch.matmul(Q, K.transpose(-2, -1)) / sqrt(d_k)
# Apply padding mask before softmax
scores = scores.masked_fill(padding_mask == 0, float('-inf'))
attention_weights = F.softmax(scores, dim=-1)

Key Takeaway: Padding masks ensure that meaningless padding tokens receive zero attention weight by setting their scores to negative infinity before softmax, preventing them from corrupting your model's learned representations.