Course contentsShow
← Machine Learning and Deep Learning
Lesson 1571 of 3,53834. Diffusion ModelsFree lesson

Cross-Attention for Text Conditioning

Using cross-attention layers in the U-Net to inject text embeddings into the generation process.

Cross-Attention for Text Conditioning

What you'll learn: How cross-attention layers inject text information into diffusion models to guide image generation.

The Core Idea

In latent diffusion models (like Stable Diffusion), we want to generate images that match text descriptions like "a cat wearing a hat." But how does the denoising U-Net "know" what text prompt to follow? The answer is cross-attention.

Think of cross-attention as a bridge between two different worlds: the evolving image features (queries) and the text embeddings (keys and values). At each layer of the U-Net, the image asks: "Which parts of the text description are relevant to what I'm currently generating?"

How It Works

Unlike self-attention (where features attend to themselves), cross-attention allows image features to query the text embeddings:

  1. Image features from the U-Net serve as queries (Q)
  2. Text embeddings (from CLIP or T5) provide keys (K) and values (V)
  3. The attention mechanism computes: Attention(Q, K, V) = softmax(QK^T / √d) V

This lets each spatial location in the feature map "look up" relevant semantic information from the text prompt. When generating "a cat wearing a hat," pixels forming the hat region will attend strongly to the "hat" tokens in the text.

Why It's Powerful

Cross-attention enables fine-grained control: different parts of the image can focus on different parts of the prompt. This is why diffusion models can handle complex, compositional descriptions so well—each region independently consults the text.

Key Takeaway: Cross-attention layers in the U-Net create a dynamic lookup mechanism where spatial image features query text embeddings, allowing precise text-guided generation at every denoising step.