Course contentsShow
← Machine Learning and Deep Learning
Lesson 2683 of 3,53858. Model Compression: Pruning and DistillationFree lesson

Distilling CNNs for Image Classification

Implementing basic knowledge distillation to compress a vision model while maintaining accuracy.

Distilling CNNs for Image Classification

What You'll Learn

You'll learn how to apply knowledge distillation to compress a convolutional neural network for image classification while preserving most of its accuracy.

The Core Idea

When distilling CNNs for image classification, you train a smaller "student" network to mimic a larger, pre-trained "teacher" network. The teacher might be a ResNet-50, while the student could be a much lighter MobileNet or ResNet-18. The student learns not just from the ground-truth labels, but from the teacher's soft predictions (logits passed through temperature-scaled softmax).

Think of it like an apprentice chef learning from a master: instead of just memorizing recipes (hard labels), the apprentice watches how the master judges each dish's qualities (soft probabilities across all categories). This richer signal helps the smaller model generalize better than training from scratch.

The Distillation Process

  1. Train the teacher: First, train your large CNN to high accuracy on your image dataset
  2. Generate soft targets: Pass images through the teacher with temperature T > 1 to get smoothed probability distributions
  3. Define combined loss: The student's loss = α × distillation_loss + (1-α) × classification_loss
  4. Train the student: Optimize the student network using both the teacher's soft targets and true labels

The distillation loss typically uses KL divergence between teacher and student soft predictions, while classification loss is standard cross-entropy with hard labels. The temperature softens both distributions during distillation, then you set T=1 at inference.

Why It Works for Vision

CNNs learn hierarchical feature representations. The teacher's soft targets encode inter-class relationships (e.g., "this husky image has some wolf-like qualities"), which helps the compact student learn meaningful visual features faster and more efficiently than training alone.

Key Takeaway: Knowledge distillation lets you compress large CNNs into deployable models by training a small student network on both true labels and the teacher's soft, information-rich predictions.