Conditioning Mechanisms in Latent Diffusion
What you'll learn: How to steer a latent diffusion model's generation using text prompts, class labels, or other control signals.
The Core Idea
In previous lessons, you learned that latent diffusion models denoise random noise in a compressed latent space to generate images. But how do you tell the model what to generate? That's where conditioning mechanisms come in.
Conditioning means injecting additional information—like a text description ("a cat wearing sunglasses"), a class label (dog vs. cat), or even a sketch—into the denoising process. This guides the model to generate images matching your specifications rather than random samples.
How Conditioning Works
The denoising network in latent diffusion already takes two inputs: the noisy latent vector and the timestep. Conditioning adds a third stream of information.
Common approaches:
-
Cross-attention layers: Text embeddings (from models like CLIP) are fed into cross-attention mechanisms within the denoising U-Net. At each layer, the network "attends" to relevant words when deciding how to denoise specific image regions.
-
Concatenation: Class labels or embeddings can be concatenated directly to the latent representation or to intermediate feature maps.
-
Adaptive normalization: Conditioning signals modulate normalization layer parameters (like scaling and shifting), allowing the condition to influence processing at multiple depths.
An Analogy
Think of an artist (the denoising network) refining a blurry sketch. Without conditioning, they improve it based on general artistic knowledge. With conditioning—like you showing them reference photos and descriptions—they refine it toward your specific vision.
Key Takeaway: Conditioning mechanisms inject guidance signals (text, labels, sketches) into the latent diffusion denoising process, typically through cross-attention or adaptive normalization, enabling controlled generation of specific content.