Course contentsShow
← Machine Learning and Deep Learning
Lesson 2682 of 3,53858. Model Compression: Pruning and DistillationFree lesson

Temperature Hyperparameter in Distillation

How temperature controls the smoothness of probability distributions and affects knowledge transfer quality.

Temperature Hyperparameter in Distillation

What You'll Learn

You'll discover how the temperature parameter fine-tunes the "softness" of probability distributions during knowledge distillation, directly impacting how much information flows from teacher to student.

The Core Idea

Temperature (T) is a scalar value that controls how smooth or peaked the teacher model's output probabilities become. When you divide logits by T before applying softmax, higher temperatures "flatten" the distribution, revealing more information about wrong-but-plausible classes. Lower temperatures sharpen it, making predictions more confident.

The mechanism: Without temperature (T=1), a teacher might output [0.9, 0.05, 0.04, 0.01] for four classes. With T=5, the same logits become something like [0.5, 0.25, 0.20, 0.05]—much softer. This softer distribution exposes the teacher's "dark knowledge": which wrong answers it considers close calls.

Why It Matters for Knowledge Transfer

  • Higher T (e.g., 3-20): Creates smooth distributions that reveal subtle similarities between classes. The student learns richer inter-class relationships (e.g., "cats are more like dogs than cars"). Better knowledge transfer, especially early in training.

  • Lower T (approaching 1): Distributions become sharper, closer to hard labels. Less nuanced information, but can help stabilize training or when teacher confidence is meaningful.

  • During inference: Always use T=1 (standard softmax) for both models.

Analogy: High temperature is like a teacher explaining "why" an answer is right and others are close. Low temperature is like just stating the answer confidently without elaboration.

Key Takeaway: The temperature hyperparameter controls how much "dark knowledge" is exposed in soft targets—higher temperatures smooth distributions to reveal richer inter-class relationships, while the same temperature must be applied to both teacher and student outputs during distillation training.