Temperature Hyperparameter in Distillation
What You'll Learn
You'll discover how the temperature parameter fine-tunes the "softness" of probability distributions during knowledge distillation, directly impacting how much information flows from teacher to student.
The Core Idea
Temperature (T) is a scalar value that controls how smooth or peaked the teacher model's output probabilities become. When you divide logits by T before applying softmax, higher temperatures "flatten" the distribution, revealing more information about wrong-but-plausible classes. Lower temperatures sharpen it, making predictions more confident.
The mechanism: Without temperature (T=1), a teacher might output [0.9, 0.05, 0.04, 0.01] for four classes. With T=5, the same logits become something like [0.5, 0.25, 0.20, 0.05]—much softer. This softer distribution exposes the teacher's "dark knowledge": which wrong answers it considers close calls.
Why It Matters for Knowledge Transfer
-
Higher T (e.g., 3-20): Creates smooth distributions that reveal subtle similarities between classes. The student learns richer inter-class relationships (e.g., "cats are more like dogs than cars"). Better knowledge transfer, especially early in training.
-
Lower T (approaching 1): Distributions become sharper, closer to hard labels. Less nuanced information, but can help stabilize training or when teacher confidence is meaningful.
-
During inference: Always use T=1 (standard softmax) for both models.
Analogy: High temperature is like a teacher explaining "why" an answer is right and others are close. Low temperature is like just stating the answer confidently without elaboration.
Key Takeaway: The temperature hyperparameter controls how much "dark knowledge" is exposed in soft targets—higher temperatures smooth distributions to reveal richer inter-class relationships, while the same temperature must be applied to both teacher and student outputs during distillation training.