Course contentsShow
← Machine Learning and Deep Learning
Lesson 1218 of 3,53827. Pretrained Language Models: GPT Family and BeyondFree lesson

T5 Pretraining: Span Corruption Objective

Learn how T5 uses span corruption masking during pretraining to learn bidirectional and generative capabilities.

T5 Pretraining: Span Corruption Objective

You'll learn how T5's unique pretraining approach masks consecutive token spans and trains the model to reconstruct them, enabling both understanding and generation.

The Core Idea

Unlike BERT's single-token masking or GPT's pure next-token prediction, T5 uses span corruption during pretraining. Here's how it works:

The model randomly selects contiguous spans of tokens in the input text and replaces each entire span with a single sentinel token (like <extra_id_0>, <extra_id_1>, etc.). The model must then predict all the corrupted spans in order.

How Span Corruption Works

Think of it like a fill-in-the-blank exercise where multiple phrases are removed:

Original text: "The cat sat on the mat and slept."

Corrupted input: "The cat <extra_id_0> the mat <extra_id_1>"

Target output: "<extra_id_0> sat on <extra_id_1> and slept <extra_id_2>"

Notice how spans (not just single tokens) are replaced, and the model must generate the missing pieces in sequence. This combines:

  • Bidirectional understanding (like BERT) by seeing context on both sides of corrupted spans
  • Generative capability (like GPT) by producing multi-token outputs autoregressively

Why This Matters

Span corruption is more efficient than single-token masking—roughly 15% of tokens are corrupted by masking fewer, longer spans. It also better prepares T5's encoder-decoder architecture for real text-to-text tasks where outputs often involve generating multiple tokens, not just predicting one word at a time.

The sentinel tokens act as placeholders that organize the reconstruction task, making it clear which spans go where.

Key Takeaway: T5's span corruption objective teaches the model to understand context bidirectionally while generating multi-token sequences, perfectly aligning pretraining with its text-to-text framework for downstream tasks.