Course contentsShow
← Machine Learning and Deep Learning
Lesson 2692 of 3,53858. Model Compression: Pruning and DistillationFree lesson

Practical Distillation: Hyperparameters and Pitfalls

Loss weight balancing, temperature tuning, student capacity selection, and common failure modes.

Practical Distillation: Hyperparameters and Pitfalls

What you'll learn: How to tune critical distillation hyperparameters and avoid common failure modes that can sabotage your compression efforts.

The Core Challenge

Knowledge distillation isn't plug-and-play. Success requires balancing multiple competing objectives and selecting the right student architecture. Getting these wrong means either losing accuracy or failing to compress meaningfully.

Loss Weight Balancing

Your distillation loss combines two terms: the soft target loss (learning from teacher) and hard target loss (learning from ground truth labels). The balance matters enormously.

Analogy: Think of learning to cook. You're watching a master chef (teacher) while also reading the recipe (ground truth). Too much focus on either source hurts—you need both perspectives weighted appropriately.

Common weight ranges: distillation loss gets 0.5–0.9 weight, hard loss gets the remainder. Start with equal weighting (0.5 each), then tune based on validation performance.

Temperature Tuning

The temperature parameter (covered earlier) typically ranges from 1 to 20. Higher temperatures reveal more teacher knowledge but can destabilize training. Lower temperatures are safer but transfer less nuance.

Pitfall: Using temperature 1 wastes distillation's power—you're barely softening the targets. Start around 3–5 for image tasks, 1–3 for NLP.

Student Capacity Selection

Your student must be large enough to learn but small enough to matter. Too tiny? It can't capture teacher knowledge. Too large? Compression gains disappear.

Rule of thumb: Aim for 20–50% of teacher parameters. A 100M parameter teacher might distill well into a 20–40M student.

Common Failure Modes

  • Capacity mismatch: Student too small loses 10%+ accuracy
  • Temperature too high: Training diverges or converges to poor solutions
  • Ignoring hard targets: Student forgets actual task objectives
  • Single-shot distillation: Often iterative distillation or ensemble teachers work better

Key Takeaway: Successful distillation requires careful tuning of loss weights (typically 0.5–0.9 for soft targets), temperature (3–5 for vision), and student size (20–50% of teacher), while avoiding extremes that cause training collapse or minimal compression.