Course contentsShow
← Machine Learning and Deep Learning
Lesson 1106 of 3,53824. The Transformer ArchitectureFree lesson

Modern Encoder-Decoder Variants

How T5, BART, and mT5 adapt the encoder-decoder architecture for different tasks and scales.

Modern Encoder-Decoder Variants

What you'll learn: How state-of-the-art models like T5, BART, and mT5 adapt the classic Transformer encoder-decoder architecture for different tasks and scales.

The Evolution Beyond the Original

Remember the original Transformer encoder-decoder? Modern variants keep that foundation but make strategic modifications to optimize for specific use cases. Think of it like car manufacturers taking a proven engine design and tuning it differently for sports cars, trucks, or economy vehicles.

Key Modern Variants

T5 (Text-to-Text Transfer Transformer)

T5 treats every NLP task as text generation. Classification? Generate the label as text. Translation? Generate the target language. This unified approach means the same encoder-decoder architecture handles all tasks. T5 also introduced span corruption pre-training: mask random chunks of text and teach the model to reconstruct them.

BART (Bidirectional and Auto-Regressive Transformer)

BART combines the bidirectional encoder (processes full context like BERT) with an autoregressive decoder (generates left-to-right). It's pre-trained by corrupting text in various ways (deleting tokens, shuffling sentences) and learning to reconstruct the original. This makes BART particularly strong at generation tasks like summarization.

mT5 (Multilingual T5)

mT5 extends T5's text-to-text framework to 100+ languages simultaneously. Same architecture, but trained on multilingual data so one model handles translation, question-answering, and summarization across language boundaries.

What They Share

All three maintain the encoder-decoder split with cross-attention, but vary in:

  • Pre-training objectives (what corruptions they learn from)
  • Scale (billions of parameters)
  • Language coverage (monolingual vs multilingual)

Key Takeaway: Modern encoder-decoder variants adapt the original Transformer architecture through different pre-training strategies and scale, making them specialists for unified task handling (T5), generation quality (BART), or multilingual transfer (mT5).