Course contentsShow
← Machine Learning and Deep Learning
Lesson 1222 of 3,53827. Pretrained Language Models: GPT Family and BeyondFree lesson

BART Pretraining Corruption Schemes

Learn BART's diverse corruption strategies including token masking, deletion, infilling, and sentence permutation.

BART Pretraining Corruption Schemes

You'll learn how BART uses multiple creative text corruption strategies during pretraining to build a robust encoder-decoder model.

The Core Idea

While T5 uses span corruption and BERT uses masked language modeling, BART takes a "kitchen sink" approach: it deliberately corrupts input text in multiple different ways during pretraining, then trains the model to reconstruct the original. Think of it like training someone to be a master restorer by showing them paintings damaged in different ways—water damage, scratches, missing sections, scrambled pieces—so they learn to fix any kind of damage.

The Five Corruption Strategies

BART's denoising autoencoder uses these diverse corruption schemes:

1. Token Masking

Like BERT, randomly replace tokens with [MASK] symbols. The model learns to fill in blanks.

2. Token Deletion

Completely remove random tokens from the input. Unlike masking, the model must figure out where content is missing and what it should be.

3. Text Infilling

Replace spans of text (multiple consecutive tokens) with a single [MASK] token. The model must generate variable-length output, learning span-level reconstruction.

4. Sentence Permutation

Shuffle the order of sentences in a document. The model learns document-level structure and coherence.

5. Document Rotation

Rotate the document so it starts at a random token, and the model must identify the true beginning. This teaches positional awareness.

Why Multiple Schemes Matter

Each corruption type forces BART to learn different aspects: token-level semantics, syntactic structure, discourse coherence, and positional reasoning. This diversity makes BART exceptionally versatile for downstream tasks like summarization, translation, and generation.

Key Takeaway: BART's strength comes from training with diverse text corruptions—masking, deletion, infilling, sentence shuffling, and rotation—making it a robust encoder-decoder model for multiple generation tasks.