The Denoising Network Architecture
What you'll learn: How U-Net with time embeddings became the standard architecture for predicting noise in diffusion models.
The Core Challenge
In diffusion models, the reverse process needs a neural network that can look at a noisy image and predict what noise was added to it. But here's the twist: the network must do this for images at different noise levels. An image that's 10% noisy requires different denoising than one that's 90% corrupted.
Why U-Net?
The U-Net architecture emerged as the go-to choice for this task. Originally designed for medical image segmentation, U-Net has a distinctive structure:
- Encoder path: Gradually downsamples the input, extracting hierarchical features
- Decoder path: Upsamples back to original resolution
- Skip connections: Copy features from encoder to decoder at matching resolutions
Think of it like restoring an old photograph: you need both the big picture (what object this is) and fine details (texture, edges). Skip connections let the network preserve spatial details while understanding global context.
Time Embeddings: The Secret Sauce
The crucial innovation for diffusion models: time embeddings. The network receives:
- The noisy image
- A timestep value (e.g., t=500 out of 1000 steps)
The timestep gets embedded (similar to positional encodings in Transformers) and injected into the U-Net at multiple layers. This tells the network "how noisy" the input is, so it can adapt its denoising strategy accordingly.
Analogy: Like a photo editor where you tell the software not just what to fix, but also how damaged the photo is.
Key Takeaway: U-Net with time embeddings provides the ideal architecture for diffusion models, combining spatial understanding through skip connections with timestep-aware denoising through injected time embeddings.