Course contentsShow
← Machine Learning and Deep Learning
Lesson 2681 of 3,53858. Model Compression: Pruning and DistillationFree lesson

The Distillation Loss Function

Combining distillation loss (teacher-student KL divergence) with ground truth loss for student training.

The Distillation Loss Function

What you'll learn: How to construct the hybrid loss function that trains a student network to mimic both its teacher and the ground truth labels.

The Core Idea

In knowledge distillation, we want our student model to learn two things simultaneously: (1) match the teacher's soft probability distributions (which contain rich information about class similarities), and (2) predict the correct labels. The distillation loss function is simply a weighted combination of these two objectives.

The Mathematical Form

The complete distillation loss looks like this:

L_total = α * L_distillation + (1 - α) * L_student

Where:

  • L_distillation: The KL divergence between teacher's and student's soft outputs (both at temperature T)
  • L_student: The standard cross-entropy loss between student predictions and ground truth labels
  • α: A hyperparameter (typically 0.5 to 0.9) balancing the two objectives

Why Both Components Matter

Think of it like learning to paint from a master artist. You study both the master's technique and brushstrokes (distillation loss) and compare your work to the reference photo (ground truth loss). The master teaches you nuances; the photo keeps you accurate.

The ground truth loss prevents the student from drifting too far from correctness, while the distillation loss transfers the teacher's learned representations and relative confidence levels across classes.

Key Takeaway: The distillation loss function combines KL divergence (teacher-student agreement) with standard cross-entropy (ground truth accuracy), weighted by α, to train a compact student model that balances mimicking the teacher with learning correct predictions.