Born-Again Networks and Multi-Generation Distillation
What you'll learn: How iterative distillation creates chains of student models that can match or exceed their teachers' performance.
The Core Idea
Born-Again Networks (BANs) take knowledge distillation beyond a single teacher-student pair. Instead of stopping after one distillation step, you iteratively use each student as the teacher for the next generation. Student₁ learns from Teacher, then Student₁ becomes Teacher₂ for Student₂, and so on.
Surprisingly, this multi-generation approach often improves performance at each step, even when all models share the same architecture. It's as if teaching forces the model to organize its knowledge more clearly, benefiting subsequent learners.
Why This Works
Think of it like a teaching chain in academia: a professor teaches a graduate student, who then teaches undergraduates. The act of teaching forces the graduate student to crystallize and restructure their understanding, sometimes revealing insights the original professor missed.
In BANs, each generation:
- Inherits refined soft targets with clearer decision boundaries
- Benefits from accumulated knowledge across generations
- Can discover better local minima than random initialization
Multi-Generation Training Process
- Train Teacher (Generation 0) normally on hard labels
- Train Student₁ (Generation 1) using Teacher's soft targets
- Train Student₂ (Generation 2) using Student₁'s soft targets
- Repeat for N generations
Each generation typically uses the same architecture, making this distinct from traditional teacher-to-smaller-student distillation. The temperature parameter and distillation loss function remain consistent across generations.
When to Use It
Multi-generation distillation works best when:
- You want to squeeze maximum performance from a fixed architecture
- Training time isn't severely constrained (you're training multiple sequential models)
- The original teacher hasn't perfectly converged to the optimal solution
Key Takeaway: Born-Again Networks iteratively distill knowledge across generations of identically-sized models, with each student-turned-teacher refining the knowledge representation and often achieving better performance than its predecessor.