Course contentsShow
← Machine Learning and Deep Learning
Lesson 1568 of 3,53834. Diffusion ModelsFree lesson

Diffusion Process in Latent Space

How forward and reverse diffusion operate on compressed latent representations instead of pixels.

Diffusion Process in Latent Space

What You'll Learn

You'll understand how diffusion models perform forward noise addition and reverse denoising on compact latent representations rather than directly on high-resolution pixel data.

The Core Idea

Recall that latent diffusion uses an autoencoder to compress images into a smaller latent space first. Now, instead of running the entire diffusion process (adding noise step-by-step, then denoising) on millions of pixels, we run it on the compressed latent codes.

Think of it like this: instead of slowly corrupting and then restoring a detailed blueprint (pixels), you corrupt and restore a simplified sketch (latent code). The sketch captures all the essential information but is much smaller and faster to work with.

How It Works

Forward diffusion in latent space:

  • Start with a latent representation z₀ (from the encoder)
  • Gradually add Gaussian noise over T timesteps: z₀ → z₁ → z₂ → ... → z_T
  • The noise schedule and math are identical to pixel-space DDPM, just operating on z instead of x

Reverse diffusion in latent space:

  • A U-Net denoising network learns to predict and remove noise from z_t
  • It generates clean latents step-by-step: z_T → z_{T-1} → ... → z₀
  • Finally, the autoencoder's decoder converts z₀ back to pixels

Why This Matters

Operating in latent space means:

  • Faster training and sampling (fewer dimensions to process)
  • Lower memory usage (smaller tensors)
  • Same high-quality generation (the latent space preserves semantic information)

Key Takeaway: Latent diffusion applies the same forward noise addition and reverse denoising mechanics as DDPM, but operates on compressed latent representations instead of raw pixels, dramatically improving computational efficiency while maintaining generation quality.