Training Objectives for Score-Based Models
What you'll learn: How the score matching loss across different noise levels translates into the practical weighted denoising objective used to train diffusion models.
From Score Matching to Denoising
You've learned that score-based models learn the gradient of the log-probability (the "score") at different noise levels. But how do we actually train these models in practice?
The theoretical score matching objective requires computing complex derivatives. Instead, practitioners use a much simpler weighted denoising objective that's mathematically equivalent but computationally tractable.
The Connection
Here's the key insight: predicting the score at a noise level is equivalent to predicting the noise that was added to get there.
Think of it like this: if someone shows you a blurry photo and asks "which direction would make this sharper?", that's equivalent to asking "what blur was applied?" Both questions require understanding the same underlying information.
At each noise scale t:
- Score matching loss: minimize the difference between your predicted score and the true score
- Denoising loss: minimize the difference between your predicted noise and the actual noise added
These are two sides of the same coin! The denoising formulation is simply more practical to compute.
The Weighted Part
Different noise levels contribute differently to generation quality. The "weighted" aspect means we balance the loss across time steps, typically giving more weight to intermediate noise levels where learning is most impactful. This weighting scheme ensures the model learns effectively across the entire diffusion process.
Key Takeaway: The score matching objective across noise scales is trained in practice using a simpler weighted denoising loss—instead of directly predicting score functions, we predict the noise that was added, which is mathematically equivalent but much easier to compute and optimize.