Course contentsShow
← Machine Learning and Deep Learning
Lesson 2688 of 3,53858. Model Compression: Pruning and DistillationFree lesson

Task-Specific vs Task-Agnostic Distillation

Choosing between distilling pretrained models before fine-tuning versus distilling task-specific models.

Task-Specific vs Task-Agnostic Distillation

What you'll learn: You'll understand when to distill a pretrained model before fine-tuning versus distilling an already task-specialized model, and how this choice affects performance and flexibility.

The Core Question

When applying knowledge distillation, you face a strategic decision: should you distill the large teacher model before it's fine-tuned on your specific task (task-agnostic), or after fine-tuning (task-specific)?

Task-Agnostic Distillation

In task-agnostic distillation, you compress a general-purpose pretrained model (like BERT or ResNet) into a smaller student before any task-specific fine-tuning. The student learns the teacher's broad knowledge representation.

Analogy: This is like teaching a student general engineering principles before they specialize in civil engineering.

When to use:

  • You need one compact model that handles multiple downstream tasks
  • You want maximum flexibility to fine-tune the student later
  • Your tasks share similar domains to the pretraining data

Task-Specific Distillation

In task-specific distillation, you first fine-tune the large teacher on your target task, then distill that specialized teacher into a student.

Analogy: This is like hiring an expert civil engineer to train an apprentice specifically in bridge design.

When to use:

  • You have one critical task requiring maximum accuracy
  • The task differs significantly from pretraining objectives
  • You can afford retraining a student per task

The Trade-off

Task-specific distillation typically yields higher accuracy on individual tasks because the teacher's specialized knowledge transfers more directly. Task-agnostic distillation offers greater versatility with one student serving multiple purposes, but may require more careful fine-tuning afterward.

Key Takeaway: Choose task-agnostic distillation for flexible, multi-task deployment; choose task-specific distillation when maximizing single-task performance is paramount.