Course contentsShow
← Machine Learning and Deep Learning
Lesson 1224 of 3,53827. Pretrained Language Models: GPT Family and BeyondFree lesson

Fine-Tuning Encoder-Decoder Models

Understand best practices for fine-tuning T5 and BART on downstream tasks like summarization and translation.

Fine-Tuning Encoder-Decoder Models

What You'll Learn

You'll learn best practices for adapting pretrained encoder-decoder models like T5 and BART to specific downstream tasks such as summarization and translation.

The Core Idea

Fine-tuning encoder-decoder models is like giving a multilingual translator specialized training in legal documents—they already understand language structure, but you're teaching them task-specific patterns.

Both T5 and BART come pretrained on massive text corpora, but their real power emerges when you fine-tune them on your specific task. Here's what makes this process effective:

Task Formatting

T5 requires explicit task prefixes in your input: "summarize: [document]" or "translate English to French: [text]". This text-to-text format unifies all tasks. BART doesn't use prefixes but expects standard sequence-to-sequence formatting.

Learning Rate Considerations

Use a smaller learning rate than pretraining (typically 1e-5 to 5e-5). Encoder-decoder models have already learned language understanding; you're making careful adjustments, not wholesale changes.

Training Strategy

Fine-tune the entire model rather than just adding task-specific heads. Since encoder-decoder architectures handle both understanding (encoder) and generation (decoder), both components need task adaptation. The encoder learns to extract task-relevant features while the decoder learns appropriate output patterns.

Sequence Length Management

Both models have maximum sequence limits. For summarization, truncate long documents intelligently or use sliding windows. For translation, shorter sequences typically work within limits.

Evaluation During Training

Monitor task-specific metrics (ROUGE for summarization, BLEU for translation) alongside loss. Early stopping based on validation performance prevents overfitting.

Key Takeaway: Fine-tuning encoder-decoder models requires careful task formatting, conservative learning rates, and full-model training to adapt both understanding and generation capabilities to your downstream task.