Training Latent Diffusion Models
What You'll Learn
You'll understand the two-stage training procedure that makes latent diffusion models efficient: first training a VAE to compress images, then training the diffusion model in that compressed space.
The Two-Stage Training Strategy
Latent diffusion models use a clever divide-and-conquer approach. Training happens in two completely separate phases:
Phase 1: Pretrain the Autoencoder (VAE)
First, you train a Variational Autoencoder independently on your image dataset. This VAE learns to:
- Encode: Compress images into a compact latent representation
- Decode: Reconstruct images from that latent space
You train this until it can faithfully reconstruct images with minimal loss. Think of this as creating a "compression dictionary" that the diffusion model will later use.
Phase 2: Train the Diffusion Model
Once your VAE is frozen (weights locked), you train the diffusion model entirely in latent space:
- Encode training images to latent representations using the pretrained encoder
- Apply forward diffusion (add noise) to these latent vectors, not raw pixels
- Train the denoising network to predict and remove noise at each timestep in latent space
- The decoder stays frozen—it's only used later during inference
Why This Works
This separation is brilliant because:
- The VAE reduces dimensionality (e.g., 512×512 image → 64×64 latent), making diffusion training ~64× faster
- Training happens in a perceptually-relevant space that the VAE already learned
- Each component can be trained independently and even reused
Key Takeaway: Training latent diffusion models is a two-stage process: first pretrain a VAE to compress images into latent space, then freeze it and train the diffusion model to denoise in that compressed representation.