Time Embeddings and Conditioning
What you'll learn: How diffusion models encode timestep information using sinusoidal embeddings so the denoising network knows "how noisy" the input is.
Why the Network Needs Time Information
In diffusion models, the denoising network must remove different amounts of noise depending on which timestep we're at. At timestep t=1000 (very noisy), it needs to perform aggressive denoising. At t=10 (almost clean), it should apply gentle refinement. But how does the network know what timestep it's processing?
Sinusoidal Time Embeddings
We encode the timestep t as a continuous vector using sinusoidal functions — the same technique used in Transformer positional encodings. This creates a unique, smooth representation for each timestep:
For timestep t, we compute:
sin(t / 10000^(2i/d))for even dimensionscos(t / 10000^(2i/d))for odd dimensions
where i is the dimension index and d is the embedding size (e.g., 128 or 256).
Why sinusoidal? These functions create patterns that help the network interpolate between timesteps and generalize across the noise schedule. Similar timesteps have similar embeddings.
Conditioning the Denoising Network
The time embedding vector is then conditioned into the denoising network (often a U-Net):
- Pass timestep
tthrough sinusoidal embedding layer - Often project through a small MLP to match feature dimensions
- Add or concatenate these embeddings into intermediate layers of the U-Net
This tells every layer: "You're denoising an image at noise level t."
Key Takeaway: Sinusoidal time embeddings encode timestep information as continuous vectors that condition the denoising network, enabling it to adapt its predictions based on how noisy the input is at each step of the reverse diffusion process.