Denoising Score Matching
What you'll learn: How denoising score matching trains neural networks by having them predict the score (gradient of log probability) of data that's been intentionally corrupted with noise.
The Core Problem
Remember how score matching trains models to estimate the gradient of the log probability distribution? There's a computational challenge: calculating the true score matching objective requires expensive second-order derivatives. Denoising score matching offers an elegant workaround.
The Key Insight
Instead of working with clean data directly, add noise to your data first, then train your model to predict the score of this noisy data distribution. The brilliant part? This denoising objective is mathematically equivalent to the original score matching objective, but much simpler to compute!
Here's the process:
- Take a clean data point x
- Add Gaussian noise: x̃ = x + σε where ε ~ N(0, I)
- Train your model to predict the score of the noisy distribution: ∇log p(x̃)
- The loss becomes: ||s_θ(x̃) - ∇log q(x̃|x)||²
Why This Works
When you corrupt data with noise, computing the gradient of the noisy distribution becomes tractable. You're essentially training the model to "point toward" the clean data from the noisy observation—exactly what denoising requires. This score tells you which direction moves toward higher probability (cleaner data).
Connection to Diffusion Models
This is foundational for diffusion models! They apply denoising score matching at multiple noise levels, creating a hierarchy of denoisers that can reverse the diffusion process step by step.
Key Takeaway: Denoising score matching trains models to predict scores by corrupting data with noise, making the score matching objective computationally tractable while maintaining theoretical equivalence to the original formulation.