Course contentsShow
← Machine Learning and Deep Learning
Lesson 1219 of 3,53827. Pretrained Language Models: GPT Family and BeyondFree lesson

T5 Task Prefixes and Multi-Task Training

Understand how T5 uses task prefixes to enable multi-task learning within a single model framework.

T5 Task Prefixes and Multi-Task Training

What You'll Learn

You'll understand how T5 uses simple text prefixes to turn every NLP task into the same format, enabling one model to handle translation, summarization, question answering, and more.

The Core Idea

T5 introduces a brilliantly simple mechanism: task prefixes. Instead of requiring different model architectures or special output layers for different tasks, T5 treats everything as text-to-text conversion and uses a prefix string to tell the model what task to perform.

Think of it like giving instructions to a universal assistant. You don't need different assistants for different jobs—you just tell the same assistant: "translate English to German:", "summarize:", or "question: What is X? context: Y" and they know what to do.

How Task Prefixes Work

During pretraining and fine-tuning, T5 prepends task-specific strings to inputs:

  • Translation: "translate English to French: The house is blue"
  • Summarization: "summarize: [long article text]"
  • Question Answering: "question: Where is Paris? context: Paris is in France"
  • Classification: "cola sentence: This syntax bad is"

The model learns to associate these prefixes with the expected output format. The prefix becomes part of the input sequence, processed by the encoder like any other tokens.

Multi-Task Training Benefits

By training on many tasks simultaneously with their respective prefixes, T5 becomes a generalist model that:

  • Shares knowledge across related tasks
  • Requires no architecture changes per task
  • Simplifies deployment (one model, many capabilities)
  • Improves sample efficiency through transfer learning

This unified framework was key to T5's strong performance across diverse benchmarks—the model learns general language understanding that benefits all tasks.

Key Takeaway: T5 task prefixes transform multi-task learning into a simple text formatting problem, enabling one encoder-decoder model to handle diverse NLP tasks by conditioning on plain-text instructions.