Latent Diffusion Model Architecture
What you'll learn: How the three key components of a Latent Diffusion Model work together to generate high-quality images efficiently.
The Complete Pipeline
A Latent Diffusion Model (LDM) isn't a single network—it's an elegant three-part system that divides generation into distinct phases:
1. VAE Encoder
The encoder compresses your input image into a compact latent representation. Think of it as creating a "blueprint" that captures the essential features while discarding redundant pixel-level details. This compression typically reduces dimensions by 4-8x, making subsequent processing much more efficient.
2. U-Net Denoiser (The Core)
This is where the magic happens—but in latent space, not pixel space. The U-Net learns to reverse the diffusion process by predicting and removing noise from latent representations. It receives:
- A noisy latent vector
- A timestep embedding (telling it "how noisy" the input is)
- Optional conditioning (text, class labels, etc.)
The U-Net's skip connections help preserve fine details while the bottleneck captures high-level structure.
3. VAE Decoder
Once the U-Net has fully denoised the latent representation (producing a clean latent vector), the decoder transforms it back into pixel space, reconstructing a high-resolution image from the blueprint.
Why This Architecture Works
By working in latent space rather than pixel space, LDMs achieve the best of both worlds: the quality of diffusion models with significantly faster training and sampling. The VAE handles the heavy lifting of compression/decompression, while the U-Net focuses on learning the semantic structure in a more efficient space.
Key Takeaway: Latent Diffusion Models combine a VAE encoder (compress to latent space), a U-Net denoiser (reverse diffusion in latent space), and a VAE decoder (reconstruct to pixels), creating an efficient pipeline for high-quality generation.