Course contentsShow
← Machine Learning and Deep Learning
Lesson 2690 of 3,53858. Model Compression: Pruning and DistillationFree lesson

Ensemble Distillation

Compressing knowledge from multiple teacher models into a single efficient student model.

Ensemble Distillation

What you'll learn: How to compress knowledge from multiple expert teacher models into a single, efficient student model.

The Core Idea

Ensemble distillation extends the knowledge distillation concept to multiple teachers. Instead of training a student model to mimic one powerful teacher, you distill knowledge from an ensemble of teacher models into a single compact student. The ensemble might consist of different architectures, models trained with different initializations, or even specialists trained on different data subsets.

Think of it like learning music from a panel of judges rather than one instructor—each teacher contributes unique insights, and the student learns a richer, more generalized representation by combining their collective wisdom.

How It Works

The student model learns from the averaged soft predictions of all teacher models. During training:

  1. Pass input through all teacher models to get their soft predictions (with temperature scaling)
  2. Average or ensemble these predictions (typically arithmetic mean)
  3. Train the student using distillation loss between student outputs and the ensemble's combined output

The ensemble's collective knowledge often captures better decision boundaries and handles uncertainty more robustly than any single teacher. The student inherits this improved performance while maintaining a small footprint suitable for deployment.

Why Ensemble Distillation?

Ensembles typically outperform individual models but are too expensive for production (multiple forward passes, N× memory). Ensemble distillation gives you ensemble-level accuracy with single-model efficiency—the best of both worlds for real-world deployment scenarios.

Key Takeaway: Ensemble distillation compresses the collective wisdom of multiple teacher models into one student, achieving near-ensemble accuracy at single-model cost and speed.