Course contentsShow
← Machine Learning and Deep Learning
Lesson 1574 of 3,53834. Diffusion ModelsFree lesson

Training Latent Diffusion Models

Training procedure: pretraining the VAE, then training the diffusion model in latent space.

Training Latent Diffusion Models

What You'll Learn

You'll understand the two-stage training procedure that makes latent diffusion models efficient: first training a VAE to compress images, then training the diffusion model in that compressed space.

The Two-Stage Training Strategy

Latent diffusion models use a clever divide-and-conquer approach. Training happens in two completely separate phases:

Phase 1: Pretrain the Autoencoder (VAE)

First, you train a Variational Autoencoder independently on your image dataset. This VAE learns to:

  • Encode: Compress images into a compact latent representation
  • Decode: Reconstruct images from that latent space

You train this until it can faithfully reconstruct images with minimal loss. Think of this as creating a "compression dictionary" that the diffusion model will later use.

Phase 2: Train the Diffusion Model

Once your VAE is frozen (weights locked), you train the diffusion model entirely in latent space:

  1. Encode training images to latent representations using the pretrained encoder
  2. Apply forward diffusion (add noise) to these latent vectors, not raw pixels
  3. Train the denoising network to predict and remove noise at each timestep in latent space
  4. The decoder stays frozen—it's only used later during inference

Why This Works

This separation is brilliant because:

  • The VAE reduces dimensionality (e.g., 512×512 image → 64×64 latent), making diffusion training ~64× faster
  • Training happens in a perceptually-relevant space that the VAE already learned
  • Each component can be trained independently and even reused

Key Takeaway: Training latent diffusion models is a two-stage process: first pretrain a VAE to compress images into latent space, then freeze it and train the diffusion model to denoise in that compressed representation.