Numerical Solvers for Sampling
What you'll learn: How different numerical methods trade off speed and accuracy when simulating the reverse diffusion process for sample generation.
Why Numerical Solvers Matter
In diffusion models, sampling requires solving either a reverse-time stochastic differential equation (SDE) or a probability flow ordinary differential equation (ODE) backward through time. These equations don't have closed-form solutions, so we need numerical methods to approximate them step-by-step.
Think of it like navigating from a noisy image back to a clear one: you can take many tiny, careful steps (slow but accurate) or fewer, larger leaps (fast but less precise).
The Euler-Maruyama Method
The Euler-Maruyama solver is the simplest approach for SDEs. It's essentially the stochastic version of Euler's method:
x_{t-Δt} = x_t + drift(x_t, t) · Δt + diffusion(x_t, t) · √Δt · z
where z is random noise. This is what ancestral sampling in DDPM uses implicitly. It's straightforward but requires many small timesteps for accuracy.
Higher-Order Solvers
Higher-order methods like Heun's method, Runge-Kutta solvers, or the DPM-Solver evaluate the model multiple times per step to estimate trajectories more accurately. They can:
- Take larger timesteps while maintaining quality
- Generate samples in 10-50 steps instead of 1000
- Trade computation per step for fewer total steps
For ODEs (probability flow), deterministic solvers like RK45 are often used since there's no stochastic component.
The Speed-Quality Trade-off
Euler-Maruyama is simple and stable but slow. Advanced solvers generate comparable quality in 10× fewer steps, dramatically speeding up sampling—critical for practical applications.
Key Takeaway: Numerical solvers determine how efficiently you can sample from diffusion models by approximating the reverse SDE/ODE; higher-order methods enable faster generation with fewer evaluation steps.