Truncation and Padding Strategies
What you'll learn: How to handle variable-length text sequences efficiently by cutting them down or filling them up to uniform sizes.
Why Sequences Need Fixed Lengths
Neural networks (especially transformers) work best with batches of uniform-length inputs. But real text varies wildly—some sentences have 5 tokens, others have 500. Truncation and padding solve this mismatch by standardizing sequence lengths for batch processing.
Truncation: When Text Is Too Long
Max length truncation cuts sequences that exceed your model's limit (e.g., 512 tokens for BERT). You typically specify:
truncation='longest_first'— trim the longest sequence in a pairtruncation='only_first'— cut only the main textmax_length=512— your hard limit
Example analogy: Like trimming overgrown branches to fit a truck—you keep what matters most (usually the beginning) and discard the rest.
Padding: When Text Is Too Short
Padding adds special [PAD] tokens to shorter sequences so all items in a batch match length. Three strategies:
- Longest sequence padding — pad everything to match the longest sequence in that batch
- Max length padding — pad all sequences to a fixed maximum (e.g., always 512)
- Dynamic/batch padding — adjust per batch (more efficient than fixed max)
Computational Impact
- Fixed max-length padding: Wastes computation on padding tokens; slower for short texts
- Dynamic batch padding: More efficient—only processes what's needed per batch
- Truncation: May lose important information from long documents
Key Takeaway: Use dynamic batch padding (longest-in-batch) for efficiency and truncate strategically based on where your task's critical information typically appears—balance between computational cost and information preservation.