Loading...
The reverse process is where the magic happens: starting from pure noise, a neural network learns to iteratively denoise, reconstructing data step by step. This chapter covers score functions, denoising score matching, the reverse SDE, the DDPM training objective, noise prediction parameterizations, and loss weighting strategies.
The forward process systematically destroys data by adding noise. The reverse process learns to undo this destruction, transforming pure Gaussian noise into structured data samples. This is the generative core of diffusion models: if we can accurately reverse each noising step, we can generate new data by starting from random noise and iteratively denoising.
The key mathematical object is the score function ---the gradient of the log-density of the noisy data distribution at time . Anderson's theorem (1982) proves that knowing the score function at every time is sufficient to reverse the forward SDE exactly. Since we cannot compute the score analytically (it requires knowing the full data distribution), we train a neural network to approximate it.
Denoising score matching provides an elegant training objective: instead of estimating the score directly, we can equivalently train the network to predict the noise that was added to a clean sample to produce a noisy sample . This connection---score estimation equals noise prediction---is the foundation of DDPM and all modern diffusion models.
This chapter covers:
Click any topic to jump in
∇ₓ log p(x) — the key quantity defining how to reverse diffusion.
Score matching and SDE reversal
Learning the score by predicting noise — avoiding intractable normalizing constants.
The reverse process is also an SDE, determined by forward SDE and the score.
Loss and architecture
Predicting noise ε at each timestep — equivalent to score matching.
U-Net with time embedding, self-attention, and skip connections.
Uniform, SNR-weighted, min-SNR — balancing the loss across timesteps.
The score function is the gradient of the log-probability density with respect to the data . It points in the direction of increasing data likelihood---toward regions where data is more probable. For a noisy distribution at timestep , the score tells us which direction to move to make it look more like a slightly-less-noisy sample.
Key insight: The score function encodes everything needed to sample from a distribution without knowing the normalizing constant. This is critical because computing requires an intractable integral over all possible data, but the score can be estimated from samples alone.
The score function is the gradient of the log-density. At any point in data space, it gives a vector pointing toward higher probability regions. For a Gaussian , the score is , always pointing toward the mean . The magnitude is larger far from the mean (strong pull) and smaller near the mean (weak pull). This gradient field defines a vector field over the entire data space.
The score is a vector field over the entire data space. For a Gaussian , the score is , a linear restoring force toward the mean. Crucially, the score does not depend on the normalizing constant , because . This is what makes score-based methods tractable: we avoid the intractable partition function entirely.
Compute the score function for a 1D Gaussian at .
For a mixture of Gaussians, the score is a weighted average of individual component scores, where the weights are the posterior responsibilities. Near a component , the score is dominated by that component's gradient. Between components, the score interpolates, pulling toward the nearest mode. This multimodal structure is what makes score estimation challenging---the network must learn a complex, spatially-varying vector field.
For a mixture , the score is a posterior-weighted average: where are the responsibilities. Between modes, the score interpolates and can be zero at saddle points. The score field reveals the topology of the distribution: basins of attraction around modes, separatrices between them.
For a 50-50 mixture of and , what is the score at ?
At noise level , the score relates to the conditional expectation of the clean data given the noisy observation. Since , the score points from toward the posterior mean , scaled by the noise variance. This reveals that estimating the score is equivalent to estimating the denoised image ---the Bayesian optimal denoiser.
Tweedie's formula connects the score to the optimal denoiser: . Rearranging gives . This is the fundamental identity: the score equals the posterior-expected noise scaled by , making noise prediction equivalent to score estimation.
If and the model predicts given , what is the estimated score?
Show that for data with the forward process , the score of the marginal distribution satisfies in expectation over given , where is the noise used to generate .