Padding and Sequence Length Handling
What you'll learn: How to efficiently handle variable-length text sequences in batched LLM inference using padding strategies and attention masks.
The Challenge of Variable Lengths
When batching multiple inference requests together, you'll encounter sequences of different lengths—one prompt might be 50 tokens, another 200. Since GPUs process rectangular tensors, all sequences in a batch must have the same shape. This is where padding comes in.
How Padding Works
Padding fills shorter sequences with special tokens (typically [PAD] or token ID 0) to match the longest sequence in the batch:
Sequence A: "Hello world" → [101, 2023, 2088, 0, 0, 0]
Sequence B: "How are you today?" → [101, 2129, 2024, 2017, 2651, 102]
The Attention Mask Solution
Here's the crucial part: you don't want the model to actually process padding tokens—that wastes compute. Attention masks tell the model which tokens are real and which are padding:
Sequence A mask: [1, 1, 1, 0, 0, 0] # Only attend to first 3 tokens
Sequence B mask: [1, 1, 1, 1, 1, 1] # Attend to all tokens
The model multiplies attention scores by these masks, effectively ignoring padded positions.
Optimization Strategies
- Left-padding for generation: Pad on the left so real tokens align at the end (important for autoregressive decoding)
- Right-padding for classification: Standard approach for encoder models
- Smart batching: Group similar-length sequences together to minimize padding overhead
- Truncation policies: Define max lengths to prevent extremely long sequences from dominating batch size
Key Takeaway: Padding enables batching of variable-length sequences, while attention masks ensure the model ignores padding tokens—minimizing wasted computation while maintaining correctness.