Course contentsShow
← Machine Learning and Deep Learning
Lesson 2918 of 3,53863. Model Serving and Inference InfrastructureFree lesson

Variable-Length Input Batching

Handling sequences of different lengths in a batch through padding, truncation, and attention masking strategies.

Variable-Length Input Batching

What You'll Learn

You'll learn how to efficiently batch sequences of different lengths together for inference by using padding, truncation, and attention masking.

The Challenge of Mixed-Length Sequences

When serving NLP models or time-series models, incoming requests often contain sequences of varying lengths—one sentence might have 10 tokens, another 50. But neural networks expect uniform tensor shapes within a batch. You can't stack a 10-element sequence with a 50-element sequence directly.

Think of it like packing boxes: You have items of different sizes, but they all need to fit on the same truck (your batch tensor).

Core Strategies

Padding

Add special tokens (usually zeros or a <PAD> token) to shorter sequences until they match the longest sequence in the batch. This creates uniform tensor dimensions.

# Conceptual padding example
batch = [
    [1, 2, 3],           # length 3
    [4, 5, 6, 7, 8]      # length 5
]
# After padding to max length (5):
padded_batch = [
    [1, 2, 3, 0, 0],
    [4, 5, 6, 7, 8]
]

Truncation

Cut sequences that exceed a maximum length. This prevents extremely long sequences from dominating memory and compute.

Attention Masking

Create a binary mask indicating which positions are real data vs padding. The model uses this to ignore padded positions during computation, preventing them from influencing predictions.

attention_mask = [
    [1, 1, 1, 0, 0],  # first 3 positions are real
    [1, 1, 1, 1, 1]   # all 5 positions are real
]

Serving Implications

In dynamic batching scenarios (covered earlier), you group requests with similar lengths together to minimize wasted padding. This balances throughput gains from batching against the computational overhead of processing padding tokens.

Key Takeaway: Variable-length input batching requires padding shorter sequences, truncating longer ones, and using attention masks to ensure the model only processes real data—critical for efficient serving of sequence models.