Text Conditioning via Cross-Attention
What You'll Learn
You'll learn how diffusion models use cross-attention layers to weave text information into image generation, enabling powerful text-to-image synthesis.
The Core Idea
Cross-attention is the bridge that connects text descriptions to image generation in diffusion models. While self-attention lets image features communicate with each other, cross-attention allows image features to "look at" and query the text embeddings at every layer of the U-Net.
Think of it like a chef (the image generator) constantly referring to a recipe card (the text prompt) while cooking. At each step, the chef checks specific parts of the recipe relevant to what they're currently doing.
How It Works
In the U-Net architecture of a diffusion model:
- Text encoding: Your text prompt is first converted into embeddings (vectors that capture semantic meaning) using a text encoder like CLIP
- Cross-attention layers: Inserted between the convolutional and self-attention blocks in the U-Net
- Query-Key-Value mechanism:
- Image features generate queries ("What should I draw here?")
- Text embeddings provide keys and values ("Here's what the prompt says")
- Attention scores determine which text tokens influence each image region
For example, when generating "a red apple on a wooden table," the image regions being generated will attend to different text tokens—the apple pixels pay more attention to "red apple" while table regions focus on "wooden table."
This conditioning happens at multiple scales throughout the U-Net, allowing both fine details and overall composition to be guided by text.
Key Takeaway: Cross-attention layers enable diffusion models to condition image generation on text by letting image features query text embeddings at each denoising step, creating the powerful text-to-image capabilities seen in Stable Diffusion and similar models.