Conditional Generation in Diffusion Models
What you'll learn: How diffusion models can generate outputs guided by specific external information like class labels, text descriptions, or reference images.
The Core Idea
Unconditional diffusion models learn to denoise random noise into realistic samples, but they have no way to control what gets generated. Conditional diffusion models solve this by incorporating external information—called conditioning signals—into the denoising process. Think of it like giving an artist not just a blank canvas, but also a detailed description of what to paint.
The conditioning signal (a class label like "cat," a text prompt like "sunset over mountains," or even an image) guides every denoising step. The model learns to ask: "Given this noise and this condition, what should the cleaner version look like?"
How It Works
During training, the denoising network receives two inputs at each timestep:
- The noisy image at timestep t
- The conditioning information (encoded appropriately)
The network learns to predict noise (or the denoised image) while being aware of what it should be creating. At inference time, you provide your desired condition, and the model generates samples that match it.
Common Conditioning Types
- Class labels: Simple categorical information (e.g., digit 0-9 in MNIST)
- Text descriptions: Natural language encoded via models like CLIP
- Images: For tasks like super-resolution or inpainting
- Spatial maps: Like ControlNet's edge maps or segmentation masks
The conditioning mechanism varies—class labels might use learned embeddings, while text uses cross-attention layers to inject semantic information at multiple network depths.
Key Takeaway: Conditional generation transforms diffusion models from random sample generators into controllable creative tools by incorporating external guidance signals throughout the denoising process.