Soft Targets and Temperature Scaling
What you'll learn: How soft probability distributions from a teacher model provide richer training signals than hard one-hot labels.
When training a student model through knowledge distillation, you could simply use the teacher's final predictions (the class with highest probability). But that throws away valuable information! Soft targets are the full probability distribution output by the teacher model—capturing not just what the teacher predicts, but how confident it is and which alternative classes seemed plausible.
Why Soft Targets Matter
Imagine a teacher classifying dog breeds. For a tricky image, it might output: 70% Golden Retriever, 25% Labrador, 5% other breeds. This reveals relationships between classes—Golden Retrievers and Labradors are similar! A hard label (just "Golden Retriever") loses this nuance.
The student learns more from soft targets because they encode the teacher's "dark knowledge"—subtle patterns about which mistakes are more reasonable than others.
Temperature Scaling
Teacher models are often too confident (probabilities like 0.99, 0.01, 0.00...). To make soft targets more informative, we apply temperature scaling before computing the softmax:
Instead of softmax(logits), use softmax(logits / T) where T > 1.
Higher temperature (e.g., T=3 or T=5) "softens" the distribution, spreading probability more evenly across classes. This amplifies the relative differences between smaller probabilities, giving the student clearer signals about class similarities.
During distillation training, both teacher and student use the same temperature T. At inference, the student uses T=1 (standard softmax).
Key Takeaway: Soft targets with temperature scaling transfer the teacher's nuanced understanding of class relationships, providing far richer supervision than hard labels alone—this is the secret sauce that makes knowledge distillation effective.