Self-Distillation and Online Distillation
What you'll learn: How models can teach themselves or learn collaboratively during training without relying on a pre-trained teacher.
The Core Idea
Traditional knowledge distillation requires a large, pre-trained teacher model to guide a smaller student. But what if we don't have a teacher ready? Self-distillation and online distillation flip this paradigm: the model learns from its own predictions or from peers being trained simultaneously.
Self-Distillation
In self-distillation, a model acts as both teacher and student. During training, the model uses its own soft predictions (often from earlier checkpoints or deeper layers) as targets to refine itself. Think of it as a musician recording themselves playing, then listening back to self-correct and improve.
How it works: You might train a network in stages, where epoch t's model teaches epoch t+1's model. Or use auxiliary classifiers at intermediate layers—deeper layers distill knowledge to shallower ones within the same architecture.
Online Distillation
Online distillation involves multiple models (often identically structured) training together as peers. Instead of one being a frozen teacher, they all learn simultaneously and exchange knowledge through their soft predictions. Imagine a study group where everyone learns together, sharing insights in real-time rather than one expert lecturing.
Implementation approach: Train two or more networks in parallel. At each batch, compute the distillation loss between their predictions (using temperature scaling and soft targets as before), alongside the standard task loss. Each network benefits from the ensemble knowledge of its peers.
Why Use These Approaches?
- No pre-trained teacher required: Saves computational cost
- Improved generalization: Self-supervision often acts as regularization
- Collaborative learning: Peer networks can explore different parts of the loss landscape
Key Takeaway: Self-distillation and online distillation enable knowledge transfer without separate teacher training—models learn from themselves or each other during joint optimization, improving efficiency and sometimes performance.