Course contentsShow
← Machine Learning and Deep Learning
Lesson 1599 of 3,53834. Diffusion ModelsFree lesson

Progressive Distillation

Iteratively halving the number of sampling steps through progressive teacher-student distillation.

Progressive Distillation

What you'll learn: How to iteratively halve the number of sampling steps in diffusion models by progressively training student models that mimic teacher models.

The Core Idea

Progressive distillation is a clever technique to dramatically speed up diffusion model sampling without sacrificing quality. Instead of trying to jump directly from, say, 1000 steps to 10 steps, you take a staged approach: train a student model to mimic a teacher's 2-step output in 1 step, then repeat this process iteratively.

Here's how it works:

  1. Stage 1: Your original diffusion model (teacher) takes 1000 steps to denoise. Train a student model to produce the same result in 500 steps by learning to match what the teacher would produce after 2 consecutive denoising steps.

  2. Stage 2: The 500-step student becomes the new teacher. Train another student to do 250 steps by again learning to skip every other step.

  3. Continue: Repeat this halving process: 250 → 125 → ~64 → 32 → 16 → 8 → 4 steps.

The key insight: Each student only needs to learn a small modification from its teacher (doubling the step size), making training stable and preserving quality. By the end, you've reduced sampling from 1000 steps to perhaps just 4 or 8 steps—a 100-250× speedup!

Why It Works

Think of it like learning to skip rungs on a ladder. First, you learn to skip every other rung safely. Once that's comfortable, you learn to skip two at a time. Trying to skip ten rungs immediately would be dangerous, but building up progressively makes it achievable.

Key Takeaway: Progressive distillation iteratively halves sampling steps through staged teacher-student training, achieving massive speedups while maintaining generation quality by making each distillation step manageable.