Course contentsShow
← Machine Learning and Deep Learning
Lesson 1223 of 3,53827. Pretrained Language Models: GPT Family and BeyondFree lesson

BART vs T5: Key Architectural Differences

Compare design philosophies, pretraining objectives, and use cases between BART and T5 encoder-decoder models.

BART vs T5: Key Architectural Differences

You'll learn how BART and T5 differ in their design philosophy, pretraining approach, and intended use cases despite both being encoder-decoder models.

Core Philosophy

While both BART and T5 use encoder-decoder architectures, they diverge significantly in their design thinking:

BART (Bidirectional and Auto-Regressive Transformers) is fundamentally a denoising autoencoder. Think of it as learning to restore corrupted paintings—it sees various types of damage and learns to reconstruct the original. BART's encoder is bidirectional (sees full context) while its decoder is autoregressive (generates left-to-right).

T5 (Text-to-Text Transfer Transformer) treats every NLP task as text generation. It's like a universal translator that converts any input format into an output format, always using the same "text-in, text-out" framework.

Architectural Distinctions

  • BART: Uses standard Transformer encoder-decoder with ReLU activations, closely following the original architecture
  • T5: Employs relative position embeddings (rather than absolute) and layer normalization differences, exploring architectural variations systematically

Pretraining Objectives

  • BART: Multiple corruption schemes (token masking, deletion, infilling, sentence permutation, document rotation)—training on diverse noise patterns
  • T5: Primarily span corruption with task prefixes—more standardized, text-to-text format

Use Case Strengths

BART excels at:

  • Text generation and summarization
  • Tasks requiring reconstruction abilities

T5 excels at:

  • Multi-task learning with consistent formatting
  • Transfer learning across diverse tasks via prefixes like "translate:", "summarize:"

Key Takeaway: BART focuses on robust denoising through diverse corruption, while T5 unifies all tasks into a consistent text-to-text framework—both effective encoder-decoder models with different philosophical approaches to pretraining and task handling.