Adversarial Diffusion Distillation
What You'll Learn
You'll learn how combining GAN-style adversarial losses with diffusion distillation creates models that can generate high-quality images in just one or a few steps.
The Core Problem and Solution
Remember that diffusion models produce amazing quality but require many sampling steps (50-1000), while distillation methods like Progressive Distillation or Consistency Models reduce these steps but sometimes sacrifice quality. What if we could get the best of both worlds?
Adversarial Diffusion Distillation (ADD) merges two powerful ideas:
- Distillation from diffusion models (like you've learned)
- Adversarial training from GANs (discriminator-based losses)
How It Works
The key insight: train a student model not just to match the teacher's outputs (distillation loss), but also to fool a discriminator that judges whether samples look "real" or "generated."
The training combines:
- Distillation loss: Student learns to mimic the multi-step teacher in fewer steps
- Adversarial loss: Discriminator pushes the student to generate perceptually realistic images
- Optional score matching loss: Maintains alignment with the original diffusion score function
Think of it like learning to paint quickly: a teacher shows you the slow, careful technique (distillation), while an art critic constantly judges your fast sketches for realism (adversarial feedback). The combination forces you to be both fast AND convincing.
The Payoff
ADD enables 1-4 step generation with quality approaching or matching the original 50+ step diffusion model. This is significantly faster than standard diffusion while maintaining the high fidelity that made diffusion models popular—combining diffusion's mode coverage with GANs' perceptual sharpness.
Key Takeaway: Adversarial Diffusion Distillation combines GAN discriminators with diffusion distillation to achieve high-quality generation in very few steps, merging the strengths of both paradigms for practical, fast sampling.