Course contentsShow
← Machine Learning and Deep Learning
Lesson 1562 of 3,53834. Diffusion ModelsFree lesson

Training Objectives for Score-Based Models

Connect score matching loss across noise scales to the weighted denoising objective used in practice.

Training Objectives for Score-Based Models

What you'll learn: How the score matching loss across different noise levels translates into the practical weighted denoising objective used to train diffusion models.

From Score Matching to Denoising

You've learned that score-based models learn the gradient of the log-probability (the "score") at different noise levels. But how do we actually train these models in practice?

The theoretical score matching objective requires computing complex derivatives. Instead, practitioners use a much simpler weighted denoising objective that's mathematically equivalent but computationally tractable.

The Connection

Here's the key insight: predicting the score at a noise level is equivalent to predicting the noise that was added to get there.

Think of it like this: if someone shows you a blurry photo and asks "which direction would make this sharper?", that's equivalent to asking "what blur was applied?" Both questions require understanding the same underlying information.

At each noise scale t:

  • Score matching loss: minimize the difference between your predicted score and the true score
  • Denoising loss: minimize the difference between your predicted noise and the actual noise added

These are two sides of the same coin! The denoising formulation is simply more practical to compute.

The Weighted Part

Different noise levels contribute differently to generation quality. The "weighted" aspect means we balance the loss across time steps, typically giving more weight to intermediate noise levels where learning is most impactful. This weighting scheme ensures the model learns effectively across the entire diffusion process.

Key Takeaway: The score matching objective across noise scales is trained in practice using a simpler weighted denoising loss—instead of directly predicting score functions, we predict the noise that was added, which is mathematically equivalent but much easier to compute and optimize.