Course contentsShow
← Machine Learning and Deep Learning
Lesson 2691 of 3,53858. Model Compression: Pruning and DistillationFree lesson

Measuring Distillation Effectiveness

Evaluating compression ratio, accuracy retention, inference speedup, and knowledge transfer quality.

Measuring Distillation Effectiveness

What you'll learn: How to quantitatively evaluate whether your distilled model successfully captures the teacher's knowledge while achieving real-world deployment benefits.

After distilling a student model from a teacher, you need objective metrics to determine if the compression was worthwhile. This evaluation spans multiple dimensions beyond just accuracy.

Core Evaluation Dimensions

Compression Ratio measures how much smaller your student became. Calculate it as teacher_parameters / student_parameters or compare model file sizes. A 10× compression ratio means your student has 1/10th the parameters.

Accuracy Retention compares student vs teacher performance on your test set. If the teacher achieves 92% accuracy and the student achieves 90%, you've retained ~97.8% of the performance. Task-specific metrics (F1, BLEU, mAP) matter here depending on your domain.

Inference Speedup measures real-world deployment gains. Time both models on your target hardware (CPU, GPU, mobile). A 3× speedup means the student produces predictions three times faster—critical for production systems.

Knowledge Transfer Quality goes deeper than raw accuracy. Use techniques like:

  • Prediction agreement rate: How often do teacher and student predict the same class?
  • Soft label similarity: Compare the full probability distributions using KL divergence or cosine similarity
  • Feature representation alignment: If you used feature-based distillation, measure how closely intermediate representations match

The Holistic View

Think of this like evaluating a car engine replacement: compression ratio is size reduction, accuracy retention is horsepower preserved, speedup is fuel efficiency gained, and knowledge transfer quality checks if it still "feels" like the original.

The best distilled models achieve high compression and speedup while minimizing accuracy loss—that's the sweet spot for deployment.

Key Takeaway: Effective distillation evaluation requires measuring compression ratio, accuracy retention, inference speedup, and knowledge transfer quality together—no single metric tells the complete story of whether your student successfully learned from its teacher.