T5 Pretraining: Span Corruption Objective
You'll learn how T5's unique pretraining approach masks consecutive token spans and trains the model to reconstruct them, enabling both understanding and generation.
The Core Idea
Unlike BERT's single-token masking or GPT's pure next-token prediction, T5 uses span corruption during pretraining. Here's how it works:
The model randomly selects contiguous spans of tokens in the input text and replaces each entire span with a single sentinel token (like <extra_id_0>, <extra_id_1>, etc.). The model must then predict all the corrupted spans in order.
How Span Corruption Works
Think of it like a fill-in-the-blank exercise where multiple phrases are removed:
Original text: "The cat sat on the mat and slept."
Corrupted input: "The cat <extra_id_0> the mat <extra_id_1>"
Target output: "<extra_id_0> sat on <extra_id_1> and slept <extra_id_2>"
Notice how spans (not just single tokens) are replaced, and the model must generate the missing pieces in sequence. This combines:
- Bidirectional understanding (like BERT) by seeing context on both sides of corrupted spans
- Generative capability (like GPT) by producing multi-token outputs autoregressively
Why This Matters
Span corruption is more efficient than single-token masking—roughly 15% of tokens are corrupted by masking fewer, longer spans. It also better prepares T5's encoder-decoder architecture for real text-to-text tasks where outputs often involve generating multiple tokens, not just predicting one word at a time.
The sentinel tokens act as placeholders that organize the reconstruction task, making it clear which spans go where.
Key Takeaway: T5's span corruption objective teaches the model to understand context bidirectionally while generating multi-token sequences, perfectly aligning pretraining with its text-to-text framework for downstream tasks.