BART: Denoising Autoencoder for Pretraining
What you'll learn: How BART combines the best of both worlds—BERT's bidirectional understanding and GPT's autoregressive generation—into one powerful sequence-to-sequence model.
The Core Idea
BART (Bidirectional and Auto-Regressive Transformers) is like having a skilled editor who first reads an entire corrupted document to understand context (bidirectionally), then reconstructs it word-by-word from left to right (autoregressively). Unlike BERT (encoder-only) or GPT (decoder-only), BART uses a full encoder-decoder architecture.
How BART Works
The architecture has two components working together:
Bidirectional Encoder
The encoder reads corrupted input text and builds rich contextual representations, similar to BERT. It can look at words both before and after any position to understand meaning fully.
Autoregressive Decoder
The decoder reconstructs the original text one token at a time, left-to-right, like GPT. It uses causal attention masking and can only see previously generated tokens during reconstruction.
The Denoising Objective
BART's pretraining corrupts text using various "noising" strategies:
- Token masking (like BERT)
- Token deletion
- Sentence permutation (shuffling sentences)
- Document rotation
- Text infilling (replacing spans with mask tokens)
The model learns to recover the original, uncorrupted text—hence "denoising autoencoder."
Why This Matters
This hybrid design makes BART exceptionally versatile. The bidirectional encoder excels at understanding, while the autoregressive decoder handles generation naturally. This makes BART ideal for sequence-to-sequence tasks like summarization, translation, and question answering—tasks requiring both comprehension and generation.
Key Takeaway: BART's encoder-decoder architecture with denoising pretraining combines bidirectional context understanding with autoregressive generation, making it a powerful unified model for both understanding and generating text.