Attention Transfer and Relational Knowledge
What you'll learn: How to distill not just what a teacher network knows, but where it looks and how features relate to each other.
Beyond Raw Output Distillation
You've learned to distill soft targets and intermediate features. But there's richer information hiding in teacher networks: attention maps (which spatial regions the network focuses on) and relational structures (how features interact with each other).
What Are Attention Maps?
In CNNs, attention maps reveal which spatial locations a network considers important. For a convolutional layer producing feature maps of shape [C, H, W], we can compute attention as:
- Spatial attention: Sum across channels → shape
[H, W]heatmap - Channel attention: Aggregate spatial dimensions → shape
[C]importance weights
Rather than forcing the student to mimic exact feature activations, we transfer these attention patterns—teaching the student to "look where the teacher looks."
Relational Knowledge
Relational distillation captures how features relate to each other within a batch or layer. For example:
- Compute pairwise similarities between feature vectors
- Create a Gramian matrix capturing feature co-activations
- Preserve the correlation structure rather than absolute values
This teaches the student the geometry of the feature space—how different examples or channels interact—making the knowledge transfer more robust and generalizable.
Why This Matters
Attention and relational knowledge are often more transferable than raw activations. A smaller student may not replicate exact feature magnitudes, but it can learn to focus on the same regions and maintain similar feature relationships—preserving the teacher's reasoning process rather than just its outputs.
Key Takeaway: Attention transfer distills where a teacher focuses, while relational knowledge distills how features interact—capturing higher-order patterns beyond raw activations for more effective compression.