Course contentsShow
← Machine Learning and Deep Learning
Lesson 1220 of 3,53827. Pretrained Language Models: GPT Family and BeyondFree lesson

T5 Model Variants and Scaling

Compare T5 variants from Small to XXL, including mT5 for multilingual applications and Flan-T5 for instruction tuning.

T5 Model Variants and Scaling

You'll learn how T5 comes in different sizes from Small to XXL, and about specialized variants like mT5 and Flan-T5.

Understanding T5 Variants

Just as GPT-3 offered different model sizes (Ada, Babbage, Curie, Davinci), T5 provides a range of variants to balance performance and computational cost. The standard T5 family includes:

  • T5-Small: ~60M parameters – fastest, suitable for prototyping
  • T5-Base: ~220M parameters – good baseline performance
  • T5-Large: ~770M parameters – stronger results, moderate compute
  • T5-XL: ~3B parameters – high performance for demanding tasks
  • T5-XXL: ~11B parameters – state-of-the-art results, heavy compute

Think of these like different-sized engines: Small gets you moving quickly, while XXL delivers maximum power but requires more fuel (compute resources).

Specialized T5 Variants

mT5 (Multilingual T5) extends the text-to-text framework to 101 languages. Instead of pretraining only on English text like standard T5, mT5 learns from multilingual data, enabling cross-lingual transfer. It's perfect when your application needs to handle multiple languages with a single model.

Flan-T5 takes pretrained T5 models and further trains them with instruction tuning—exposing the model to diverse tasks phrased as natural language instructions. This makes Flan-T5 better at following user instructions and performing zero-shot tasks, similar to how InstructGPT improved GPT-3.

Choosing Your Variant

Select based on your constraints: use Small/Base for resource-limited environments or rapid iteration, Large/XL for production systems requiring quality, and XXL when performance is paramount. For multilingual needs, choose mT5. For instruction-following and zero-shot capabilities, Flan-T5 offers the best trade-off.

Key Takeaway: T5 scales from 60M to 11B parameters across five sizes, with mT5 adding multilingual support and Flan-T5 enhancing instruction-following through targeted fine-tuning.