Course contentsShow
← Machine Learning and Deep Learning
Lesson 1272 of 3,53828. NLP Preprocessing and TokenizationFree lesson

Truncation and Padding Strategies

Max length truncation, longest sequence padding, and batch padding; impact on computational efficiency.

Truncation and Padding Strategies

What you'll learn: How to handle variable-length text sequences efficiently by cutting them down or filling them up to uniform sizes.

Why Sequences Need Fixed Lengths

Neural networks (especially transformers) work best with batches of uniform-length inputs. But real text varies wildly—some sentences have 5 tokens, others have 500. Truncation and padding solve this mismatch by standardizing sequence lengths for batch processing.

Truncation: When Text Is Too Long

Max length truncation cuts sequences that exceed your model's limit (e.g., 512 tokens for BERT). You typically specify:

  • truncation='longest_first' — trim the longest sequence in a pair
  • truncation='only_first' — cut only the main text
  • max_length=512 — your hard limit

Example analogy: Like trimming overgrown branches to fit a truck—you keep what matters most (usually the beginning) and discard the rest.

Padding: When Text Is Too Short

Padding adds special [PAD] tokens to shorter sequences so all items in a batch match length. Three strategies:

  1. Longest sequence padding — pad everything to match the longest sequence in that batch
  2. Max length padding — pad all sequences to a fixed maximum (e.g., always 512)
  3. Dynamic/batch padding — adjust per batch (more efficient than fixed max)

Computational Impact

  • Fixed max-length padding: Wastes computation on padding tokens; slower for short texts
  • Dynamic batch padding: More efficient—only processes what's needed per batch
  • Truncation: May lose important information from long documents

Key Takeaway: Use dynamic batch padding (longest-in-batch) for efficiency and truncate strategically based on where your task's critical information typically appears—balance between computational cost and information preservation.