Course contentsShow
← Machine Learning and Deep Learning
Lesson 2680 of 3,53858. Model Compression: Pruning and DistillationFree lesson

Soft Targets and Temperature Scaling

How soft probability distributions from the teacher provide richer training signals than hard labels.

Soft Targets and Temperature Scaling

What you'll learn: How soft probability distributions from a teacher model provide richer training signals than hard one-hot labels.

When training a student model through knowledge distillation, you could simply use the teacher's final predictions (the class with highest probability). But that throws away valuable information! Soft targets are the full probability distribution output by the teacher model—capturing not just what the teacher predicts, but how confident it is and which alternative classes seemed plausible.

Why Soft Targets Matter

Imagine a teacher classifying dog breeds. For a tricky image, it might output: 70% Golden Retriever, 25% Labrador, 5% other breeds. This reveals relationships between classes—Golden Retrievers and Labradors are similar! A hard label (just "Golden Retriever") loses this nuance.

The student learns more from soft targets because they encode the teacher's "dark knowledge"—subtle patterns about which mistakes are more reasonable than others.

Temperature Scaling

Teacher models are often too confident (probabilities like 0.99, 0.01, 0.00...). To make soft targets more informative, we apply temperature scaling before computing the softmax:

Instead of softmax(logits), use softmax(logits / T) where T > 1.

Higher temperature (e.g., T=3 or T=5) "softens" the distribution, spreading probability more evenly across classes. This amplifies the relative differences between smaller probabilities, giving the student clearer signals about class similarities.

During distillation training, both teacher and student use the same temperature T. At inference, the student uses T=1 (standard softmax).

Key Takeaway: Soft targets with temperature scaling transfer the teacher's nuanced understanding of class relationships, providing far richer supervision than hard labels alone—this is the secret sauce that makes knowledge distillation effective.