Autoencoder Component of Latent Diffusion
What you'll learn: How a VAE encoder-decoder compresses images into compact latent representations that make diffusion models faster and more practical.
Why Compress Before Diffusing?
You've learned that diffusion models work beautifully in pixel space (DDPM), but there's a problem: images are huge. A 512×512 RGB image contains 786,432 numbers! Running diffusion for hundreds of steps on that many pixels is computationally expensive.
Latent Diffusion solves this by first compressing images into a much smaller latent representation using a Variational Autoencoder (VAE), then performing diffusion in that compact space.
The VAE Encoder-Decoder Architecture
Think of the VAE as a compression-decompression machine:
-
Encoder: Takes a high-resolution image and squeezes it down to a small latent code (e.g., 64×64×4 instead of 512×512×3). This captures the essential "meaning" of the image in far fewer numbers.
-
Decoder: Takes that compact latent code and reconstructs the full image back to its original resolution.
The encoder learns to throw away unimportant details (like exact pixel positions of tiny textures) while preserving semantic information (like "there's a dog in the center"). The decoder learns to fill in those details plausibly during reconstruction.
Why This Matters for Diffusion
By working in latent space, the diffusion model:
- Operates on 8× or more smaller representations
- Runs much faster (fewer pixels to denoise)
- Still generates high-quality images (the decoder handles fine details)
Key Takeaway: The VAE encoder-decoder in Latent Diffusion compresses images into compact latent codes where diffusion happens efficiently, then decodes them back to full resolution—making high-quality generation practical.