Course contentsShow
← Machine Learning and Deep Learning
Lesson 1572 of 3,53834. Diffusion ModelsFree lesson

Stable Diffusion Architecture Overview

The complete Stable Diffusion pipeline: CLIP text encoder, U-Net, and VAE decoder.

Stable Diffusion Architecture Overview

What you'll learn: How Stable Diffusion assembles three key components—CLIP text encoder, U-Net, and VAE decoder—into a complete text-to-image pipeline.

The Three-Part Pipeline

Stable Diffusion isn't a single model, but rather an elegant combination of three specialized neural networks working together:

1. CLIP Text Encoder

This component transforms your text prompt (like "a cat wearing a hat") into a numerical representation that captures semantic meaning. CLIP creates embeddings that the model can understand and use for conditioning.

2. U-Net (The Denoising Core)

This is where the diffusion magic happens. Operating in latent space (not raw pixels), the U-Net iteratively removes noise from a random latent representation. It uses cross-attention to incorporate the text embeddings from CLIP at each denoising step, ensuring the generated image matches your prompt.

3. VAE Decoder

Once the U-Net completes denoising in latent space, the VAE (Variational Autoencoder) decoder upsamples this compact representation back into a full-resolution image you can see.

How They Work Together

Think of it like a factory assembly line:

  1. Text → Meaning: CLIP translates your words into concept vectors
  2. Noise → Structure: U-Net sculpts random noise into organized latent features, guided by those concepts
  3. Latent → Pixels: VAE decoder renders the latent code into a beautiful image

This architecture is why Stable Diffusion generates high-quality images much faster than pixel-space diffusion models—the U-Net works with compressed latent representations (roughly 8× smaller), making each denoising step computationally cheaper.

Key Takeaway: Stable Diffusion chains CLIP text encoding, latent-space U-Net denoising, and VAE decoding into an efficient text-to-image pipeline that balances quality with computational speed.