Course contentsShow
← Machine Learning and Deep Learning
Lesson 1569 of 3,53834. Diffusion ModelsFree lesson

Latent Diffusion Model Architecture

Complete architecture: VAE encoder, U-Net denoiser in latent space, and VAE decoder.

Latent Diffusion Model Architecture

What you'll learn: How the three key components of a Latent Diffusion Model work together to generate high-quality images efficiently.

The Complete Pipeline

A Latent Diffusion Model (LDM) isn't a single network—it's an elegant three-part system that divides generation into distinct phases:

1. VAE Encoder

The encoder compresses your input image into a compact latent representation. Think of it as creating a "blueprint" that captures the essential features while discarding redundant pixel-level details. This compression typically reduces dimensions by 4-8x, making subsequent processing much more efficient.

2. U-Net Denoiser (The Core)

This is where the magic happens—but in latent space, not pixel space. The U-Net learns to reverse the diffusion process by predicting and removing noise from latent representations. It receives:

  • A noisy latent vector
  • A timestep embedding (telling it "how noisy" the input is)
  • Optional conditioning (text, class labels, etc.)

The U-Net's skip connections help preserve fine details while the bottleneck captures high-level structure.

3. VAE Decoder

Once the U-Net has fully denoised the latent representation (producing a clean latent vector), the decoder transforms it back into pixel space, reconstructing a high-resolution image from the blueprint.

Why This Architecture Works

By working in latent space rather than pixel space, LDMs achieve the best of both worlds: the quality of diffusion models with significantly faster training and sampling. The VAE handles the heavy lifting of compression/decompression, while the U-Net focuses on learning the semantic structure in a more efficient space.

Key Takeaway: Latent Diffusion Models combine a VAE encoder (compress to latent space), a U-Net denoiser (reverse diffusion in latent space), and a VAE decoder (reconstruct to pixels), creating an efficient pipeline for high-quality generation.