Week 3-4

Chapter 4: Reverse Process & Denoising

The reverse process is where the magic happens: starting from pure noise, a neural network learns to iteratively denoise, reconstructing data step by step. This chapter covers score functions, denoising score matching, the reverse SDE, the DDPM training objective, noise prediction parameterizations, and loss weighting strategies.

Chapter Overview

The forward process systematically destroys data by adding noise. The reverse process learns to undo this destruction, transforming pure Gaussian noise into structured data samples. This is the generative core of diffusion models: if we can accurately reverse each noising step, we can generate new data by starting from random noise and iteratively denoising.

The key mathematical object is the score function $\nabla_x \log p_t(x)$ ---the gradient of the log-density of the noisy data distribution at time $t$ . Anderson's theorem (1982) proves that knowing the score function at every time $t$ is sufficient to reverse the forward SDE exactly. Since we cannot compute the score analytically (it requires knowing the full data distribution), we train a neural network to approximate it.

Denoising score matching provides an elegant training objective: instead of estimating the score directly, we can equivalently train the network to predict the noise $\epsilon$ that was added to a clean sample $x_0$ to produce a noisy sample $x_t$ . This connection---score estimation equals noise prediction---is the foundation of DDPM and all modern diffusion models.

This chapter covers:

Score Functions: The gradient of the log-density and its role in generative modeling
Denoising Score Matching: Training neural networks to estimate scores via noise prediction
The Reverse SDE: Anderson's theorem and the reverse-time diffusion process
DDPM Training Objective: The simplified loss function that makes training practical
Parameterizing the Noise Predictor: $\epsilon$ -prediction vs $x_0$ -prediction vs $v$ -prediction
Loss Weighting Strategies: How different timestep weightings affect sample quality

Chapter Roadmap

Click any topic to jump in

Score Functions

∇ₓ log p(x) — the key quantity defining how to reverse diffusion.

Definition and Geometric MeaningScore of a Gaussian MixtureScore for Noisy Distributions

Learning the reverse

Score matching and SDE reversal

Denoising Score Matching

Learning the score by predicting noise — avoiding intractable normalizing constants.

The Score Matching ObjectiveDenoising Score Matching TheoremFrom Score Matching to Noise Prediction

Reverse SDE

The reverse process is also an SDE, determined by forward SDE and the score.

Anderson's Reverse-Time SDEThe Probability Flow ODEDiscretizing the Reverse Process

Training in practice

Loss and architecture

DDPM Objective

Predicting noise ε at each timestep — equivalent to score matching.

The Variational Lower BoundSimplified Training LossVLB vs Simple Loss in Practice

Noise Predictor

U-Net with time embedding, self-attention, and skip connections.

Epsilon-PredictionX0-PredictionV-Prediction

Optimizing the objective

Loss Weighting

Uniform, SNR-weighted, min-SNR — balancing the loss across timesteps.

SNR-Based WeightingMin-SNR Loss WeightingPerceptual Loss Weighting

The score function $s(x) = \nabla_x \log p(x)$ is the gradient of the log-probability density with respect to the data $x$ . It points in the direction of increasing data likelihood---toward regions where data is more probable. For a noisy distribution $p_t(x)$ at timestep $t$ , the score $\nabla_x \log p_t(x)$ tells us which direction to move $x$ to make it look more like a slightly-less-noisy sample.

Key insight: The score function encodes everything needed to sample from a distribution without knowing the normalizing constant. This is critical because computing $p(x)$ requires an intractable integral over all possible data, but the score $\nabla_x \log p(x)$ can be estimated from samples alone.

In this topic

1Definition and Geometric Meaning

2Score of a Gaussian Mixture

3Score for Noisy Distributions

1 of 3

Definition and Geometric Meaning

$s(x) = \nabla_x \log p(x) = \frac{\nabla_x p(x)}{p(x)}$

The score function is the gradient of the log-density. At any point $x$ in data space, it gives a vector pointing toward higher probability regions. For a Gaussian $p(x) = \mathcal{N}(x; \mu, \sigma^2 I)$ , the score is $s(x) = -(x - \mu)/\sigma^2$ , always pointing toward the mean $\mu$ . The magnitude $\|s(x)\|$ is larger far from the mean (strong pull) and smaller near the mean (weak pull). This gradient field defines a vector field over the entire data space.

Mathematical Intuition

The score $s(x) = \nabla_x \log p(x) = \nabla_x p(x)/p(x)$ is a vector field over the entire data space. For a Gaussian $\mathcal{N}(\mu, \sigma^2 I)$ , the score is $-(x-\mu)/\sigma^2$ , a linear restoring force toward the mean. Crucially, the score does not depend on the normalizing constant $Z = \int p^*(x)dx$ , because $\nabla_x \log(p^*/Z) = \nabla_x \log p^*$ . This is what makes score-based methods tractable: we avoid the intractable partition function entirely.

Example:

Compute the score function for a 1D Gaussian $p(x) = \mathcal{N}(x; 3, 4)$ at $x = 7$ .

2 of 3

Score of a Gaussian Mixture

$\nabla_x \log p(x) = \frac{\sum_k \pi_k \mathcal{N}(x; \mu_k, \sigma_k^2)\, \frac{-(x - \mu_k)}{\sigma_k^2}}{\sum_k \pi_k \mathcal{N}(x; \mu_k, \sigma_k^2)}$

For a mixture of Gaussians, the score is a weighted average of individual component scores, where the weights are the posterior responsibilities. Near a component $k$ , the score is dominated by that component's gradient. Between components, the score interpolates, pulling toward the nearest mode. This multimodal structure is what makes score estimation challenging---the network must learn a complex, spatially-varying vector field.

Mathematical Intuition

For a mixture $p(x) = \sum_k \pi_k \mathcal{N}(x;\mu_k,\sigma_k^2 I)$ , the score is a posterior-weighted average: $\nabla_x \log p = \sum_k r_k(x) \cdot (-(x-\mu_k)/\sigma_k^2)$ where $r_k(x) = \pi_k \mathcal{N}(x;\mu_k,\sigma_k^2)/p(x)$ are the responsibilities. Between modes, the score interpolates and can be zero at saddle points. The score field reveals the topology of the distribution: basins of attraction around modes, separatrices between them.

Example:

For a 50-50 mixture of $\mathcal{N}(-3, 1)$ and $\mathcal{N}(3, 1)$ , what is the score at $x = 0$ ?

3 of 3

Score for Noisy Distributions

$\nabla_{x_t} \log p_t(x_t) = -\frac{x_t - \sqrt{\bar{\alpha}_t}\, \mathbb{E}[x_0 | x_t]}{1 - \bar{\alpha}_t}$

At noise level $t$ , the score relates to the conditional expectation of the clean data given the noisy observation. Since $x_t = \sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \epsilon$ , the score points from $x_t$ toward the posterior mean $\sqrt{\bar{\alpha}_t}\, \mathbb{E}[x_0 | x_t]$ , scaled by the noise variance. This reveals that estimating the score is equivalent to estimating the denoised image $\mathbb{E}[x_0 | x_t]$ ---the Bayesian optimal denoiser.

Mathematical Intuition

Tweedie's formula connects the score to the optimal denoiser: $\mathbb{E}[x_0|x_t] = (x_t + (1-\bar{\alpha}_t)\nabla_{x_t} \log p_t(x_t))/\sqrt{\bar{\alpha}_t}$ . Rearranging gives $\nabla_{x_t} \log p_t = -(x_t - \sqrt{\bar{\alpha}_t}\mathbb{E}[x_0|x_t])/(1-\bar{\alpha}_t) = -\mathbb{E}[\epsilon|x_t]/\sqrt{1-\bar{\alpha}_t}$ . This is the fundamental identity: the score equals the posterior-expected noise scaled by $-1/\sqrt{1-\bar{\alpha}_t}$ , making noise prediction equivalent to score estimation.

Example:

If $\bar{\alpha}_t = 0.5$ and the model predicts $\hat{x}_0 = [2.0, -1.0]$ given $x_t = [1.5, 0.3]$ , what is the estimated score?

Theory Exercise

Problem:

Show that for data $x_0$ with the forward process $x_t = \sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \epsilon$ , the score of the marginal distribution $p_t(x_t)$ satisfies $\nabla_{x_t} \log p_t(x_t) = -\epsilon / \sqrt{1 - \bar{\alpha}_t}$ in expectation over $x_0$ given $x_t$ , where $\epsilon$ is the noise used to generate $x_t$ .

Hints:

Start from $q(x_t | x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t}\, x_0, (1 - \bar{\alpha}_t) I)$
The score of a Gaussian $\mathcal{N}(\mu, \sigma^2 I)$ is $-(x - \mu)/\sigma^2$
Use Tweedie's formula: $\mathbb{E}[x_0 | x_t] = (x_t + (1 - \bar{\alpha}_t) \nabla_{x_t} \log p_t(x_t)) / \sqrt{\bar{\alpha}_t}$

Papers

Score-Based Generative Modeling through Stochastic Differential Equations Generative Modeling by Estimating Gradients of the Data Distribution

Blogs

Generative Modeling by Estimating Gradients of the Data Distribution — Yang Song What are Diffusion Models? — Lilian Weng

Forward Diffusion Process

Sampling & Acceleration

Chapter 4: Reverse Process & Denoising

Chapter Overview

Chapter Roadmap

Score Functions

Denoising Score Matching

Reverse SDE

DDPM Objective

Noise Predictor

Loss Weighting

Score Functions

In this topic

Definition and Geometric Meaning

Score of a Gaussian Mixture

Score for Noisy Distributions

Theory Exercise

Problem:

Hints:

Denoising Score Matching

The Reverse SDE

DDPM Training Objective

Parameterizing the Noise Predictor

Loss Weighting Strategies