Feature-Based Distillation
What You'll Learn
You'll learn how to transfer knowledge from a teacher to a student model by matching internal layer representations, not just final outputs.
Beyond Output Mimicry
So far, distillation has focused on making the student mimic the teacher's soft output predictions (softmax probabilities). But a deep network learns rich intermediate representations at each layer—feature maps that capture edges, textures, patterns, and high-level concepts.
Feature-based distillation extends knowledge transfer by forcing the student's internal layers to produce similar feature maps to the teacher's corresponding layers.
Core Idea
Think of learning to paint from a master artist. You could just try to match their final painting (output distillation). But you'd learn faster if you also matched their sketch lines, color blocking, and shading techniques—the intermediate steps that lead to the final result.
Similarly, we add loss terms that measure the difference between teacher and student features at selected layers:
Loss = Classification Loss + Distillation Loss + Feature Loss
Where the feature loss might be:
Feature Loss = ||f_teacher(x) - f_student(x)||²
Here, f(x) represents the activation maps from a chosen intermediate layer.
Why It Works
Intermediate features encode hierarchical knowledge: low-level edges, mid-level textures, high-level semantics. By aligning these, the student learns how the teacher processes information, not just what it predicts. This often yields better generalization, especially when the student architecture differs significantly from the teacher.
You typically need projection layers if the teacher and student have different channel dimensions at the matching point.
Key Takeaway:
Feature-based distillation transfers knowledge by matching intermediate layer representations between teacher and student, teaching the student not just what to predict, but how to think.