Diffusion Process in Latent Space
What You'll Learn
You'll understand how diffusion models perform forward noise addition and reverse denoising on compact latent representations rather than directly on high-resolution pixel data.
The Core Idea
Recall that latent diffusion uses an autoencoder to compress images into a smaller latent space first. Now, instead of running the entire diffusion process (adding noise step-by-step, then denoising) on millions of pixels, we run it on the compressed latent codes.
Think of it like this: instead of slowly corrupting and then restoring a detailed blueprint (pixels), you corrupt and restore a simplified sketch (latent code). The sketch captures all the essential information but is much smaller and faster to work with.
How It Works
Forward diffusion in latent space:
- Start with a latent representation
z₀(from the encoder) - Gradually add Gaussian noise over
Ttimesteps:z₀ → z₁ → z₂ → ... → z_T - The noise schedule and math are identical to pixel-space DDPM, just operating on
zinstead ofx
Reverse diffusion in latent space:
- A U-Net denoising network learns to predict and remove noise from
z_t - It generates clean latents step-by-step:
z_T → z_{T-1} → ... → z₀ - Finally, the autoencoder's decoder converts
z₀back to pixels
Why This Matters
Operating in latent space means:
- Faster training and sampling (fewer dimensions to process)
- Lower memory usage (smaller tensors)
- Same high-quality generation (the latent space preserves semantic information)
Key Takeaway: Latent diffusion applies the same forward noise addition and reverse denoising mechanics as DDPM, but operates on compressed latent representations instead of raw pixels, dramatically improving computational efficiency while maintaining generation quality.