Course contentsShow
← AI Engineering
Lesson 1021 of 1,88625. Model Serving and Inference OptimizationFree lesson

Padding and Sequence Length Handling

Manage variable-length sequences in batches using padding strategies and attention masks to minimize wasted computation.

Padding and Sequence Length Handling

What you'll learn: How to efficiently handle variable-length text sequences in batched LLM inference using padding strategies and attention masks.

The Challenge of Variable Lengths

When batching multiple inference requests together, you'll encounter sequences of different lengths—one prompt might be 50 tokens, another 200. Since GPUs process rectangular tensors, all sequences in a batch must have the same shape. This is where padding comes in.

How Padding Works

Padding fills shorter sequences with special tokens (typically [PAD] or token ID 0) to match the longest sequence in the batch:

Sequence A: "Hello world"          → [101, 2023, 2088, 0, 0, 0]
Sequence B: "How are you today?"   → [101, 2129, 2024, 2017, 2651, 102]

The Attention Mask Solution

Here's the crucial part: you don't want the model to actually process padding tokens—that wastes compute. Attention masks tell the model which tokens are real and which are padding:

Sequence A mask: [1, 1, 1, 0, 0, 0]  # Only attend to first 3 tokens
Sequence B mask: [1, 1, 1, 1, 1, 1]  # Attend to all tokens

The model multiplies attention scores by these masks, effectively ignoring padded positions.

Optimization Strategies

  • Left-padding for generation: Pad on the left so real tokens align at the end (important for autoregressive decoding)
  • Right-padding for classification: Standard approach for encoder models
  • Smart batching: Group similar-length sequences together to minimize padding overhead
  • Truncation policies: Define max lengths to prevent extremely long sequences from dominating batch size

Key Takeaway: Padding enables batching of variable-length sequences, while attention masks ensure the model ignores padding tokens—minimizing wasted computation while maintaining correctness.