Course contentsShow
← Machine Learning and Deep Learning
Lesson 1032 of 3,53823. Recurrent Neural NetworksFree lesson

Loss Functions for Sequence Generation

Computing cross-entropy loss over variable-length target sequences with proper masking for padding tokens.

Loss Functions for Sequence Generation

What you'll learn: How to properly compute cross-entropy loss for variable-length sequences while ignoring padding tokens.

The Challenge with Variable-Length Outputs

When training sequence-to-sequence models (like machine translation or text generation), your target sequences often have different lengths. You pad shorter sequences to create uniform batches, but here's the problem: you don't want your model to learn from those padding tokens!

Think of it like grading essays of different lengths. You wouldn't penalize a student for "mistakes" on blank pages you added to make all submissions the same thickness.

Cross-Entropy Loss with Masking

For each position in your sequence, you compute cross-entropy loss between the predicted token distribution and the actual target token. However, you must:

  1. Apply a mask to identify which positions contain real tokens vs. padding
  2. Zero out the loss at padded positions
  3. Normalize by the number of real (non-padded) tokens, not the total padded length

The formula looks like:

Loss = -∑(mask[t] × log P(y_t | context)) / ∑mask[t]

where mask[t] is 1 for real tokens and 0 for padding.

Why This Matters

Without proper masking, your model wastes capacity learning to predict padding tokens and gets misleading loss values. A short accurate sequence and a long inaccurate one might show similar losses if padding dominates the calculation.

Most frameworks (PyTorch, TensorFlow) provide built-in support for masked loss computation, often through an ignore_index parameter that automatically handles this.

Key Takeaway: When training sequence generators, always mask out padding tokens in your loss calculation and normalize by actual sequence length to ensure your model learns only from real data.