Handling Different Sequence Lengths
What you'll learn: How to efficiently batch sequences of different lengths together for training recurrent neural networks.
The Challenge
When training RNNs on real-world data, you'll rarely have sequences of identical length. One sentence might have 5 words, another 20. But neural networks expect fixed-size batches for efficient parallel processing on GPUs. How do you combine these varying-length sequences into a single batch?
Three Key Techniques
Padding
Think of padding like adding blank pages to books of different lengths so they all fit the same-size bookshelf. You add special "padding" tokens (usually zeros) to shorter sequences until all sequences in a batch match the longest one's length.
Example: Sequences [1,2,3] and [4,5,6,7,8] become [1,2,3,0,0] and [4,5,6,7,8].
Masking
Padding creates "fake" timesteps that shouldn't influence your model. Masking tells the model which positions are real data and which are padding. During loss calculation, masked positions are ignored—like crossing out those blank pages so they don't count toward your grade.
Packing
This advanced technique removes padding altogether for computational efficiency. You pack only the real sequence elements into a condensed format, along with metadata about original lengths. The RNN processes only valid timesteps, then unpacks results back to padded form when needed.
Why It Matters
Without these techniques, you'd either:
- Process one sequence at a time (painfully slow)
- Truncate all sequences to the shortest one (losing information)
- Waste computation on meaningless padding tokens
Key Takeaway: Padding standardizes batch shapes, masking prevents padding from affecting learning, and packing optimizes computation—together enabling efficient training on variable-length sequential data.