Score Functions and the Score Matching Objective
What you'll learn: You'll understand what a score function is and how score matching offers a powerful alternative to traditional likelihood-based training for generative models.
What Is a Score Function?
A score function is simply the gradient of the log-probability density with respect to the input data. Mathematically, if you have some data distribution with density p(x), the score function is:
∇ₓ log p(x)
Think of it as an arrow field that points in the direction where the data becomes more likely. At any point in space, the score tells you which way to move to climb up the probability hill.
Why Score Matching?
Training generative models usually requires computing likelihoods, which means normalizing probability distributions—a computationally expensive or intractable step for complex models. Score matching sidesteps this entirely by learning the score function directly, without ever computing the normalization constant.
The score matching objective trains a model to predict these gradient directions. Instead of asking "what's the probability of this data?", we ask "which direction makes this data more probable?"
The Connection to Diffusion
This is crucial for diffusion models: during the reverse (denoising) process, we need to know which direction removes noise and recovers the true data distribution. The score function provides exactly this information—it guides us from noisy samples back toward high-probability, clean data.
Key Takeaway: The score function is the gradient of log-density, pointing toward higher probability regions. Score matching trains models to predict these gradients without computing expensive likelihood normalizations, making it ideal for diffusion models.