Course contentsShow
← Machine Learning and Deep Learning
Lesson 2685 of 3,53858. Model Compression: Pruning and DistillationFree lesson

Attention Transfer and Relational Knowledge

Distilling attention maps and relational structures between features rather than raw activations.

Attention Transfer and Relational Knowledge

What you'll learn: How to distill not just what a teacher network knows, but where it looks and how features relate to each other.

Beyond Raw Output Distillation

You've learned to distill soft targets and intermediate features. But there's richer information hiding in teacher networks: attention maps (which spatial regions the network focuses on) and relational structures (how features interact with each other).

What Are Attention Maps?

In CNNs, attention maps reveal which spatial locations a network considers important. For a convolutional layer producing feature maps of shape [C, H, W], we can compute attention as:

  • Spatial attention: Sum across channels → shape [H, W] heatmap
  • Channel attention: Aggregate spatial dimensions → shape [C] importance weights

Rather than forcing the student to mimic exact feature activations, we transfer these attention patterns—teaching the student to "look where the teacher looks."

Relational Knowledge

Relational distillation captures how features relate to each other within a batch or layer. For example:

  • Compute pairwise similarities between feature vectors
  • Create a Gramian matrix capturing feature co-activations
  • Preserve the correlation structure rather than absolute values

This teaches the student the geometry of the feature space—how different examples or channels interact—making the knowledge transfer more robust and generalizable.

Why This Matters

Attention and relational knowledge are often more transferable than raw activations. A smaller student may not replicate exact feature magnitudes, but it can learn to focus on the same regions and maintain similar feature relationships—preserving the teacher's reasoning process rather than just its outputs.

Key Takeaway: Attention transfer distills where a teacher focuses, while relational knowledge distills how features interact—capturing higher-order patterns beyond raw activations for more effective compression.