Padding and Truncation Strategies
What you'll learn: How to configure tokenizers to handle variable-length inputs for consistent batch processing.
Why We Need Padding and Truncation
When you tokenize text, different sentences produce different numbers of tokens. But models expect all inputs in a batch to have the same length—like requiring all runners in a race to start from the same line.
Think of it like packing suitcases: some travelers pack light (short sentences), others pack heavy (long sentences). To fit them all on the same luggage rack, you need rules: pad small cases with filler material, and trim oversized ones to fit.
The Two Key Parameters
max_length
Sets the target length for all sequences. Shorter sequences get padded up to this length; longer ones get truncated down to it.
padding and truncation
Control whether these operations happen at all:
padding=Truefills shorter sequences with special padding tokenstruncation=Truecuts longer sequences to fitmax_length
Configuration in Practice
When you tokenize, you can specify these directly:
tokenizer(
text,
max_length=128,
padding=True,
truncation=True
)
This ensures every sequence comes out exactly 128 tokens long—short ones padded, long ones trimmed. The model ignores padded positions using the attention mask you learned earlier.
Why It Matters
Without consistent shapes, you can't process multiple examples together efficiently. Padding and truncation turn messy, real-world text into neat, uniform batches the model can process in parallel.
Key Takeaway: Padding and truncation with max_length convert variable-length text into fixed-size tensors, enabling efficient batch processing while respecting model input constraints.