Loss Functions for Sequence Generation
What you'll learn: How to properly compute cross-entropy loss for variable-length sequences while ignoring padding tokens.
The Challenge with Variable-Length Outputs
When training sequence-to-sequence models (like machine translation or text generation), your target sequences often have different lengths. You pad shorter sequences to create uniform batches, but here's the problem: you don't want your model to learn from those padding tokens!
Think of it like grading essays of different lengths. You wouldn't penalize a student for "mistakes" on blank pages you added to make all submissions the same thickness.
Cross-Entropy Loss with Masking
For each position in your sequence, you compute cross-entropy loss between the predicted token distribution and the actual target token. However, you must:
- Apply a mask to identify which positions contain real tokens vs. padding
- Zero out the loss at padded positions
- Normalize by the number of real (non-padded) tokens, not the total padded length
The formula looks like:
Loss = -∑(mask[t] × log P(y_t | context)) / ∑mask[t]
where mask[t] is 1 for real tokens and 0 for padding.
Why This Matters
Without proper masking, your model wastes capacity learning to predict padding tokens and gets misleading loss values. A short accurate sequence and a long inaccurate one might show similar losses if padding dominates the calculation.
Most frameworks (PyTorch, TensorFlow) provide built-in support for masked loss computation, often through an ignore_index parameter that automatically handles this.
Key Takeaway: When training sequence generators, always mask out padding tokens in your loss calculation and normalize by actual sequence length to ensure your model learns only from real data.