The Distillation Loss Function
What you'll learn: How to construct the hybrid loss function that trains a student network to mimic both its teacher and the ground truth labels.
The Core Idea
In knowledge distillation, we want our student model to learn two things simultaneously: (1) match the teacher's soft probability distributions (which contain rich information about class similarities), and (2) predict the correct labels. The distillation loss function is simply a weighted combination of these two objectives.
The Mathematical Form
The complete distillation loss looks like this:
L_total = α * L_distillation + (1 - α) * L_student
Where:
- L_distillation: The KL divergence between teacher's and student's soft outputs (both at temperature T)
- L_student: The standard cross-entropy loss between student predictions and ground truth labels
- α: A hyperparameter (typically 0.5 to 0.9) balancing the two objectives
Why Both Components Matter
Think of it like learning to paint from a master artist. You study both the master's technique and brushstrokes (distillation loss) and compare your work to the reference photo (ground truth loss). The master teaches you nuances; the photo keeps you accurate.
The ground truth loss prevents the student from drifting too far from correctness, while the distillation loss transfers the teacher's learned representations and relative confidence levels across classes.
Key Takeaway: The distillation loss function combines KL divergence (teacher-student agreement) with standard cross-entropy (ground truth accuracy), weighted by α, to train a compact student model that balances mimicking the teacher with learning correct predictions.