Chapter 4: Reverse Process & Denoising

Week 3-4

The reverse process is where the magic happens: starting from pure noise, a neural network learns to iteratively denoise, reconstructing data step by step. This chapter covers score functions, denoising score matching, the reverse SDE, the DDPM training objective, noise prediction parameterizations, and loss weighting strategies.

Topics: 6
Demo: 1
Videos: 3
Min read: 35

Chapter Overview

The forward process systematically destroys data by adding noise. The reverse process learns to undo this destruction, transforming pure Gaussian noise into structured data samples. This is the generative core of diffusion models: if we can accurately reverse each noising step, we can generate new data by starting from random noise and iteratively denoising.

The key mathematical object is the score function $\nabla_x \log p_t(x)$ ---the gradient of the log-density of the noisy data distribution at time $t$ . Anderson's theorem (1982) proves that knowing the score function at every time $t$ is sufficient to reverse the forward SDE exactly. Since we cannot compute the score analytically (it requires knowing the full data distribution), we train a neural network to approximate it.

Denoising score matching provides an elegant training objective: instead of estimating the score directly, we can equivalently train the network to predict the noise $\epsilon$ that was added to a clean sample $x_0$ to produce a noisy sample $x_t$ . This connection---score estimation equals noise prediction---is the foundation of DDPM and all modern diffusion models.

This chapter covers:

Score Functions: The gradient of the log-density and its role in generative modeling
Denoising Score Matching: Training neural networks to estimate scores via noise prediction
The Reverse SDE: Anderson's theorem and the reverse-time diffusion process
DDPM Training Objective: The simplified loss function that makes training practical
Parameterizing the Noise Predictor: $\epsilon$ -prediction vs $x_0$ -prediction vs $v$ -prediction
Loss Weighting Strategies: How different timestep weightings affect sample quality

Chapter Roadmap

Click any topic to jump in

Explainer videoInteractive demo

Score Functions

∇ₓ log p(x) — the key quantity defining how to reverse diffusion.

Definition and Geometric MeaningScore of a Gaussian MixtureScore for Noisy Distributions

Learning the reverse

Score matching and SDE reversal

Denoising Score Matching

Learning the score by predicting noise — avoiding intractable normalizing constants.

The Score Matching ObjectiveDenoising Score Matching TheoremFrom Score Matching to Noise Prediction

Reverse SDE

The reverse process is also an SDE, determined by forward SDE and the score.

Anderson's Reverse-Time SDEThe Probability Flow ODEDiscretizing the Reverse Process

Training in practice

Loss and architecture

DDPM Objective

Predicting noise ε at each timestep — equivalent to score matching.

The Variational Lower BoundSimplified Training LossVLB vs Simple Loss in Practice

Noise Predictor

U-Net with time embedding, self-attention, and skip connections.

Epsilon-PredictionX0-PredictionV-Prediction

Optimizing the objective

Loss Weighting

Uniform, SNR-weighted, min-SNR — balancing the loss across timesteps.

SNR-Based WeightingMin-SNR Loss WeightingPerceptual Loss Weighting

Topics

6 topics in this chapter

The forward-process chapter taught us to destroy data: any sample becomes pure Gaussian noise through the reparameterized kernel $x_t = \sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \epsilon$ . Generation demands the opposite, turning noise back into data, and that reversal needs a quantity we have not yet met: the score function $s(x) = \nabla_x \log p(x)$ , the gradient of the log-density. The problem it solves is fundamental. To sample from a distribution we would normally need its density $p(x)$ , but computing $p(x)$ requires an intractable normalizing integral over all possible data. The score sidesteps this entirely: it points toward higher-probability regions, it can be estimated from samples alone, and the normalizing constant cancels out of the gradient. This topic first defines the score and its geometry for a single Gaussian, then examines the far richer multimodal structure of a Gaussian mixture, and finally connects the score at any noise level to the Bayesian optimal denoiser, the key identity that makes the entire reverse process learnable from data.

Definition

The score function $s(x) = \nabla_x \log p(x)$ is the gradient of the log-probability density with respect to $x$ . It is a vector field pointing toward regions of higher likelihood, independent of the intractable normalizing constant, so it can be estimated from data samples alone. Knowing the score at every noise level suffices to reverse the diffusion process.

In this topic

1Definition and Geometric Meaning
2Score of a Gaussian Mixture
3Score for Noisy Distributions

1 of 3

Definition and Geometric Meaning

$s(x) = \nabla_x \log p(x) = \frac{\nabla_x p(x)}{p(x)}$

The score function is the gradient of the log-density. At any point $x$ in data space, it gives a vector pointing toward higher probability regions. For a Gaussian $p(x) = \mathcal{N}(x; \mu, \sigma^2 I)$ , the score is $s(x) = -(x - \mu)/\sigma^2$ , always pointing toward the mean $\mu$ . The magnitude $\|s(x)\|$ is larger far from the mean (strong pull) and smaller near the mean (weak pull). This gradient field defines a vector field over the entire data space.

Mathematical Intuition

The score $s(x) = \nabla_x \log p(x) = \nabla_x p(x)/p(x)$ is a vector field over the entire data space. For a Gaussian $\mathcal{N}(\mu, \sigma^2 I)$ , the score is $-(x-\mu)/\sigma^2$ , a linear restoring force toward the mean. Crucially, the score does not depend on the normalizing constant $Z = \int p^*(x)dx$ , because $\nabla_x \log(p^*/Z) = \nabla_x \log p^*$ . This is what makes score-based methods tractable: we avoid the intractable partition function entirely.

Example:

Compute the score function for a 1D Gaussian $p(x) = \mathcal{N}(x; 3, 4)$ at $x = 7$ .

2 of 3

Score of a Gaussian Mixture

$\nabla_x \log p(x) = \frac{\sum_k \pi_k \mathcal{N}(x; \mu_k, \sigma_k^2)\, \frac{-(x - \mu_k)}{\sigma_k^2}}{\sum_k \pi_k \mathcal{N}(x; \mu_k, \sigma_k^2)}$

For a mixture of Gaussians, the score is a posterior-weighted average of the individual component scores, where the weights are the responsibilities, the probability that each component generated $x$ . Near component $k$ the score is dominated by that component's linear pull toward $\mu_k$ ; between components the score interpolates, steering toward whichever mode is nearest. This multimodal structure is exactly what makes score estimation hard: the network must learn a complex, spatially varying vector field with basins around every mode and unstable saddle points between them. It connects directly to the single-Gaussian case from the previous concept, which is the special one-component limit where the responsibilities collapse to one.

Mathematical Intuition

For a mixture $p(x) = \sum_k \pi_k \mathcal{N}(x;\mu_k,\sigma_k^2 I)$ , the score is a posterior-weighted average: $\nabla_x \log p = \sum_k r_k(x) \cdot (-(x-\mu_k)/\sigma_k^2)$ where $r_k(x) = \pi_k \mathcal{N}(x;\mu_k,\sigma_k^2)/p(x)$ are the responsibilities. Between modes, the score interpolates and can be zero at saddle points. The score field reveals the topology of the distribution: basins of attraction around modes, separatrices between them.

Example:

For a 50-50 mixture of $\mathcal{N}(-3, 1)$ and $\mathcal{N}(3, 1)$ , what is the score at $x = 0$ ?

3 of 3

Score for Noisy Distributions

$\nabla_{x_t} \log p_t(x_t) = -\frac{x_t - \sqrt{\bar{\alpha}_t}\, \mathbb{E}[x_0 | x_t]}{1 - \bar{\alpha}_t}$

At noise level $t$ the score relates directly to the conditional expectation of the clean data given the noisy observation. Because $x_t = \sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \epsilon$ , the score points from $x_t$ toward the posterior mean $\sqrt{\bar{\alpha}_t}\, \mathbb{E}[x_0 \mid x_t]$ , rescaled by the noise variance $1 - \bar{\alpha}_t$ . This reveals the identity at the heart of the chapter: estimating the score is equivalent to estimating the denoised sample $\mathbb{E}[x_0 \mid x_t]$ , the Bayesian optimal denoiser. The connection, formalized by Tweedie's formula, means a network trained simply to denoise implicitly learns the score, which is what the next topic exploits.

Mathematical Intuition

Tweedie's formula connects the score to the optimal denoiser: $\mathbb{E}[x_0|x_t] = (x_t + (1-\bar{\alpha}_t)\nabla_{x_t} \log p_t(x_t))/\sqrt{\bar{\alpha}_t}$ . Rearranging gives $\nabla_{x_t} \log p_t = -(x_t - \sqrt{\bar{\alpha}_t}\mathbb{E}[x_0|x_t])/(1-\bar{\alpha}_t) = -\mathbb{E}[\epsilon|x_t]/\sqrt{1-\bar{\alpha}_t}$ . This is the fundamental identity: the score equals the posterior-expected noise scaled by $-1/\sqrt{1-\bar{\alpha}_t}$ , making noise prediction equivalent to score estimation.

Example:

If $\bar{\alpha}_t = 0.5$ and the model predicts $\hat{x}_0 = [2.0, -1.0]$ given $x_t = [1.5, 0.3]$ , what is the estimated score?

Theory Exercise

Problem:

Show that for data $x_0$ with the forward process $x_t = \sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \epsilon$ , the score of the marginal distribution $p_t(x_t)$ satisfies $\nabla_{x_t} \log p_t(x_t) = -\epsilon / \sqrt{1 - \bar{\alpha}_t}$ in expectation over $x_0$ given $x_t$ , where $\epsilon$ is the noise used to generate $x_t$ .

Hints:

Start from $q(x_t | x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t}\, x_0, (1 - \bar{\alpha}_t) I)$
The score of a Gaussian $\mathcal{N}(\mu, \sigma^2 I)$ is $-(x - \mu)/\sigma^2$
Use Tweedie's formula: $\mathbb{E}[x_0 | x_t] = (x_t + (1 - \bar{\alpha}_t) \nabla_{x_t} \log p_t(x_t)) / \sqrt{\bar{\alpha}_t}$

Papers

Score-Based Generative Modeling through Stochastic Differential Equations Generative Modeling by Estimating Gradients of the Data Distribution

Blogs

Generative Modeling by Estimating Gradients of the Data Distribution — Yang Song What are Diffusion Models? — Lilian Weng

Definition

Denoising score matching trains a network to match the score of the tractable noise kernel $q(x_t \mid x_0)$ , $\nabla_{x_t} \log q(x_t \mid x_0) = -\epsilon / \sqrt{1 - \bar{\alpha}_t}$ rather than the intractable marginal score $\nabla_x \log p_t(x)$ . The two objectives share the same optimum, so a network that learns to denoise noisy samples provably recovers the true score, reducing score estimation to simple noise prediction.

In this topic

1The Score Matching Objective
2Denoising Score Matching Theorem
3From Score Matching to Noise Prediction

1 of 3

The Score Matching Objective

$\mathcal{L}_{\text{SM}} = \mathbb{E}_{p(x)}\left[\frac{1}{2}\|s_\theta(x) - \nabla_x \log p(x)\|^2\right]$

The ideal objective is to minimize the expected squared error between the model's score $s_\theta(x)$ and the true score. However, this requires knowing $\nabla_x \log p(x)$ , which is intractable. The seminal insight of score matching (Hyv"arinen, 2005) is that this objective can be rewritten using integration by parts to avoid computing the true score, but the resulting expression involves the Hessian trace of $s_\theta$ , which is expensive. Denoising score matching provides a simpler alternative.

Mathematical Intuition

The explicit score matching loss $\mathbb{E}_p[\frac{1}{2}\|s_\theta - \nabla_x \log p\|^2]$ is intractable because $\nabla_x \log p(x)$ requires knowing $p$ . Hyvärinen (2005) showed via integration by parts that this equals $\mathbb{E}_p[\frac{1}{2}\|s_\theta\|^2 + \text{tr}(\nabla_x s_\theta)] + C$ , avoiding the true score but requiring the Jacobian trace, which is $O(d^2)$ for $d$ -dimensional data. Denoising score matching avoids both issues.

Example:

Why can't we directly minimize $\|s_\theta(x) - \nabla_x \log p(x)\|^2$ using training data?

2 of 3

Denoising Score Matching Theorem

$\mathcal{L}_{\text{DSM}} = \mathbb{E}_{t, x_0, \epsilon}\left[\left\|s_\theta(x_t, t) + \frac{\epsilon}{\sqrt{1 - \bar{\alpha}_t}}\right\|^2\right]$

The denoising score matching loss is equivalent to the true score matching loss up to a constant. Instead of matching the intractable true score, we match the score of the noise kernel $q(x_t | x_0)$ , which is simply $-\epsilon/\sqrt{1 - \bar{\alpha}_t}$ . At the optimum, $s_\theta^*(x_t, t) = \nabla_{x_t} \log p_t(x_t)$ for all $x_t$ and $t$ . The proof relies on the fact that $\nabla_{x_t} \log p_t(x_t) = \mathbb{E}_{q(x_0|x_t)}[\nabla_{x_t} \log q(x_t | x_0)]$ .

Mathematical Intuition

The DSM loss $\mathbb{E}[\|s_\theta(x_t,t) + \epsilon/\sqrt{1-\bar{\alpha}_t}\|^2]$ equals the true score matching loss up to a $\theta$ -independent constant. The proof uses the tower property: $\mathbb{E}_{x_t}[s_\theta \cdot \nabla \log p_t] = \mathbb{E}_{x_t}[s_\theta \cdot \mathbb{E}_{x_0|x_t}[\nabla \log q(x_t|x_0)]] = \mathbb{E}_{x_0,x_t}[s_\theta \cdot \nabla \log q(x_t|x_0)]$ . The cross-terms match, so the objectives have the same optimizer: $s_\theta^* = \nabla_{x_t} \log p_t$ .

Example:

Show that minimizing the DSM loss over $s_\theta$ yields the true marginal score at the optimum.

3 of 3

From Score Matching to Noise Prediction

$s_\theta(x_t, t) = -\frac{\epsilon_\theta(x_t, t)}{\sqrt{1 - \bar{\alpha}_t}}$

Rather than parameterizing the score network directly, DDPM parameterizes a noise-prediction network $\epsilon_\theta(x_t, t)$ and defines the score implicitly through $s_\theta = -\epsilon_\theta / \sqrt{1 - \bar{\alpha}_t}$ . Substituting this into the denoising score matching loss collapses everything to $\mathbb{E}[\|\epsilon - \epsilon_\theta(x_t, t)\|^2]$ : just predict the noise. This reparameterization is both simpler and numerically better conditioned than regressing the score directly, because the target $\epsilon$ has unit variance at every timestep whereas the score magnitude blows up as $t \to 0$ . It is the direct payoff of the theorem in the previous concept, turning an abstract score-matching guarantee into the concrete mean-squared-error objective used in practice.

Mathematical Intuition

Reparameterizing $s_\theta(x_t,t) = -\epsilon_\theta(x_t,t)/\sqrt{1-\bar{\alpha}_t}$ absorbs the timestep-dependent scaling into the loss. The DSM loss becomes $\mathbb{E}[\|\epsilon - \epsilon_\theta(x_t,t)\|^2/(1-\bar{\alpha}_t)]$ . Dropping the $(1-\bar{\alpha}_t)^{-1}$ weighting gives the DDPM simple loss, which empirically outperforms the correctly-weighted version. This works because $\epsilon$ has unit variance at all $t$ , providing a well-conditioned regression target.

Example:

If $\epsilon_\theta$ predicts $\hat{\epsilon} = [0.8, -1.2]$ at $\bar{\alpha}_t = 0.5$ , what is the implied score estimate?

Theory Exercise

Problem:

Prove that the denoising score matching objective $\mathbb{E}_{x_0, \epsilon, t}[\|s_\theta(x_t,t) - \nabla_{x_t} \log q(x_t|x_0)\|^2]$ differs from the true score matching objective $\mathbb{E}_{x_t, t}[\|s_\theta(x_t,t) - \nabla_{x_t} \log p_t(x_t)\|^2]$ only by a constant independent of $\theta$ .

Hints:

Expand both squared norms and compare term by term
The cross-term in DSM involves $\mathbb{E}_{x_0|x_t}[\nabla_{x_t} \log q(x_t|x_0)]$
Use the identity $\nabla_{x_t} \log p_t(x_t) = \mathbb{E}_{q(x_0|x_t)}[\nabla_{x_t} \log q(x_t|x_0)]$

Papers

Denoising Diffusion Probabilistic Models Score-Based Generative Modeling through Stochastic Differential Equations

Blogs

What are Diffusion Models? — Lilian Weng Generative Modeling by Estimating Gradients of the Data Distribution — Yang Song

Definition

The reverse SDE $dx = [f(x,t) - g(t)^2 \nabla_x \log p_t(x)]\,dt + g(t)\,d\bar{w}$ runs the diffusion backward in time from noise to data. It shares the forward drift and diffusion but adds a score-correction term that steers each step toward higher-probability regions. Its only unknown is the score, which the trained network supplies, making generation possible.

In this topic

1Anderson's Reverse-Time SDE
2The Probability Flow ODE
3Discretizing the Reverse Process

1 of 3

Anderson's Reverse-Time SDE

$dx = \left[f(x, t) - g(t)^2 \nabla_x \log p_t(x)\right] dt + g(t)\, d\bar{w}$

The reverse SDE runs backward in time from $t = T$ to $t = 0$ . It has the same drift $f(x,t)$ and diffusion $g(t)$ as the forward SDE, plus a correction term $-g(t)^2 \nabla_x \log p_t(x)$ that fights against the noise injection. This correction term uses the score function to steer the process toward high-probability regions at each timestep. The $d\bar{w}$ is a reverse-time Brownian motion, which is a standard Brownian motion when time runs backward.

Mathematical Intuition

Anderson's theorem (1982) states that the reverse of $dx = f(x,t)dt + g(t)dw$ is $dx = [f - g^2 \nabla_x \log p_t]dt + g\,d\bar{w}$ . The correction term $-g^2 \nabla_x \log p_t$ exactly counteracts the entropy increase from the forward diffusion. Intuitively, the forward SDE increases entropy (adds uncertainty), while the score correction decreases entropy (adds structure). At the continuous limit, these balance to produce exact reversal.

Example:

Write the reverse SDE for the VP-SDE where $f(x,t) = -\frac{1}{2}\beta(t)x$ and $g(t) = \sqrt{\beta(t)}$ .

2 of 3

The Probability Flow ODE

$dx = \left[f(x, t) - \frac{1}{2}g(t)^2 \nabla_x \log p_t(x)\right] dt$

A deterministic counterpart to the reverse SDE, the probability flow ODE generates samples from the same distribution but without randomness. It has no $d\bar{w}$ term. The key property is that the probability flow ODE induces the same marginal distributions $p_t(x)$ at every time $t$ as the SDE. This means we can compute exact likelihoods via the change-of-variables formula (instantaneous change of variables / continuous normalizing flows). The ODE formulation also enables deterministic sampling (same initial noise always produces the same image) and more efficient solvers.

Mathematical Intuition

The probability flow ODE $dx = [f - \frac{1}{2}g^2 \nabla_x \log p_t]dt$ is a deterministic ODE that induces the same marginal densities $p_t(x)$ as the SDE at every time $t$ . By the instantaneous change of variables formula, the log-likelihood is $\log p_0(x_0) = \log p_T(x_T) + \int_0^T \text{tr}(\partial f_{\text{ODE}}/\partial x)dt$ . This enables exact likelihood computation, deterministic encoding (mapping data to latent), and efficient adaptive-step solvers.

Example:

Compare the number of function evaluations needed for the SDE vs ODE formulation with equal sample quality.

3 of 3

Discretizing the Reverse Process

$x_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{\beta_t}{\sqrt{1 - \bar{\alpha}_t}}\epsilon_\theta(x_t, t)\right) + \sigma_t z$

In practice, we discretize the reverse SDE into $T$ steps. Starting from $x_T \sim \mathcal{N}(0, I)$ , each step computes the predicted mean using the noise predictor $\epsilon_\theta$ and adds fresh Gaussian noise $z \sim \mathcal{N}(0, I)$ scaled by $\sigma_t$ . The noise scale $\sigma_t$ can be set to $\sqrt{\beta_t}$ (matching the forward process) or $\sqrt{\tilde{\beta}_t}$ where $\tilde{\beta}_t = \frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_t}\beta_t$ (the posterior variance). At $t = 1$ , no noise is added ( $\sigma_1 = 0$ ) since we want a clean output.

Mathematical Intuition

The DDPM reverse step $x_{t-1} = \frac{1}{\sqrt{\alpha_t}}(x_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\epsilon_\theta) + \sigma_t z$ is the Euler-Maruyama discretization of the reverse SDE. The mean comes from replacing the score with $-\epsilon_\theta/\sqrt{1-\bar{\alpha}_t}$ in Anderson's formula. The variance $\sigma_t^2$ can be $\beta_t$ (upper bound) or $\tilde{\beta}_t = \beta_t(1-\bar{\alpha}_{t-1})/(1-\bar{\alpha}_t)$ (posterior variance, tighter). DDIM sets $\sigma_t = 0$ , recovering the probability flow ODE.

Example:

At timestep $t = 500$ with $\alpha_t = 0.99$ , $\bar{\alpha}_t = 0.5$ , and $\epsilon_\theta(x_t, t) = [1.2, -0.8]$ , compute the predicted mean $\mu$ of $x_{t-1}$ .

Theory Exercise

Problem:

Derive the posterior distribution $q(x_{t-1} | x_t, x_0)$ for the DDPM forward process. Show that it is Gaussian with mean $\tilde{\mu}_t = \frac{\sqrt{\bar{\alpha}_{t-1}}\beta_t}{1 - \bar{\alpha}_t} x_0 + \frac{\sqrt{\alpha_t}(1 - \bar{\alpha}_{t-1})}{1 - \bar{\alpha}_t} x_t$ and variance $\tilde{\beta}_t = \frac{(1 - \bar{\alpha}_{t-1})}{(1 - \bar{\alpha}_t)} \beta_t$ .

Hints:

Apply Bayes' theorem: $q(x_{t-1}|x_t, x_0) \propto q(x_t|x_{t-1}) q(x_{t-1}|x_0)$
Both terms on the right are Gaussians; their product is Gaussian
Complete the square in the exponent to find the posterior mean and variance

Papers

Score-Based Generative Modeling through Stochastic Differential Equations Denoising Diffusion Probabilistic Models

Blogs

Generative Modeling by Estimating Gradients of the Data Distribution — Yang Song What are Diffusion Models? — Lilian Weng

Definition

The DDPM training objective is the simplified loss $L_{\text{simple}} = \mathbb{E}_{t, x_0, \epsilon}[\|\epsilon - \epsilon_\theta(x_t, t)\|^2]$ , a uniformly weighted mean-squared error on noise prediction. It arises by simplifying the variational lower bound: every per-timestep KL term reduces to this form, and discarding the theoretical weights empirically improves sample quality while making training as simple as a denoiser.

In this topic

1The Variational Lower Bound
2Simplified Training Loss
3VLB vs Simple Loss in Practice

1 of 3

The Variational Lower Bound

$\log p(x_0) \geq \mathbb{E}_q\left[-\log \frac{q(x_{1:T}|x_0)}{p_\theta(x_{0:T})}\right] = -L_{\text{VLB}}$

Like VAEs, diffusion models maximize a variational lower bound on the log-likelihood. The bound decomposes into: $L_0$ (reconstruction term at $t = 0$ ), $L_t = D_{KL}(q(x_{t-1}|x_t, x_0) \| p_\theta(x_{t-1}|x_t))$ for $t = 1, \ldots, T-1$ (denoising matching terms), and $L_T = D_{KL}(q(x_T|x_0) \| p(x_T))$ (prior matching, typically negligible). The bulk of the loss comes from the $L_t$ terms, which measure how well the model's reverse steps match the true posterior reverse steps.

Mathematical Intuition

The ELBO decomposes as $L_{\text{VLB}} = L_T + \sum_{t=2}^T L_t + L_0$ where $L_t = D_{KL}(q(x_{t-1}|x_t,x_0) \| p_\theta(x_{t-1}|x_t))$ . Since both distributions are Gaussian with the same variance $\tilde{\beta}_t$ , each $L_t = \frac{1}{2\tilde{\beta}_t}\|\tilde{\mu}_t - \mu_\theta\|^2$ . The true posterior mean $\tilde{\mu}_t$ uses the known $x_0$ ; the model mean $\mu_\theta$ uses the predicted $\hat{x}_0 = (x_t - \sqrt{1-\bar{\alpha}_t}\epsilon_\theta)/\sqrt{\bar{\alpha}_t}$ .

Example:

Why is $L_T$ negligible and typically ignored during training?

2 of 3

Simplified Training Loss

$L_{\text{simple}} = \mathbb{E}_{t \sim \mathcal{U}(1,T),\, x_0,\, \epsilon}\left[\|\epsilon - \epsilon_\theta(x_t, t)\|^2\right]$

Ho et al. showed that dropping the per-timestep weighting from the VLB and using a uniform weight over all timesteps produces better samples, despite being a looser bound on the likelihood. The simplified loss samples a random timestep $t$ uniformly, generates $x_t = \sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \epsilon$ , and trains the network to predict $\epsilon$ . This is just mean squared error on noise prediction---no KL divergences, no complex derivations at training time.

Mathematical Intuition

The simple loss $L_{\text{simple}} = \mathbb{E}_{t,x_0,\epsilon}[\|\epsilon - \epsilon_\theta(x_t,t)\|^2]$ drops the per-timestep VLB weight $w_t = \beta_t^2/(2\tilde{\beta}_t \alpha_t(1-\bar{\alpha}_t))$ , using uniform weighting instead. This is a looser bound on the likelihood but produces better FID scores. The reason: VLB over-weights low-noise steps (fine details) at the expense of high-noise steps (global structure). Uniform weighting forces the model to be accurate at all noise levels, improving overall sample coherence.

Example:

Write the pseudocode for one training step of DDPM.

3 of 3

VLB vs Simple Loss in Practice

$L_{\text{VLB}} = \sum_{t=1}^{T} w_t \cdot \mathbb{E}[\|\epsilon - \epsilon_\theta(x_t, t)\|^2], \quad w_t = \frac{\beta_t^2}{2\sigma_t^2 \alpha_t (1 - \bar{\alpha}_t)}$

The VLB loss weights each timestep $t$ by a factor proportional to $\text{SNR}(t-1) - \text{SNR}(t)$ where $\text{SNR}(t) = \bar{\alpha}_t / (1 - \bar{\alpha}_t)$ . This gives more weight to low-noise timesteps (where perceptual details matter) and less weight to high-noise timesteps. The simple loss uses uniform weighting, which empirically produces better FID scores. The intuition is that uniform weighting forces the model to allocate capacity across all noise levels, including the high-noise regime where global structure is determined. Training with VLB often produces sharper but less diverse samples.

Mathematical Intuition

The VLB weight $w_t \propto \text{SNR}(t-1) - \text{SNR}(t) = \Delta\text{SNR}(t)$ , which is the discrete derivative of the SNR schedule. For the linear schedule, $\Delta\text{SNR}$ is largest at small $t$ (low noise) where SNR changes rapidly, and smallest at large $t$ . This means VLB allocates gradient signal according to the information destruction rate. The simple loss, with $w_t = 1$ , allocates uniformly. Hybrid losses $L_{\text{simple}} + \lambda L_{\text{VLB}}$ with $\lambda \ll 1$ balance sample quality and likelihood.

Example:

At which timesteps does the VLB loss place the most weight, and why?

Theory Exercise

Problem:

Starting from the KL divergence term $L_t = D_{KL}(q(x_{t-1}|x_t, x_0) \| p_\theta(x_{t-1}|x_t))$ where both distributions are Gaussian, derive the expression in terms of the noise prediction $\epsilon_\theta$ . Show that $L_t \propto \|\epsilon - \epsilon_\theta(x_t, t)\|^2$ with a timestep-dependent weighting.

Hints:

KL between two Gaussians: $D_{KL}(\mathcal{N}(\mu_1,\sigma^2) \| \mathcal{N}(\mu_2,\sigma^2)) = \frac{1}{2\sigma^2}\|\mu_1 - \mu_2\|^2$
Both the true posterior mean $\tilde{\mu}_t$ and model mean $\mu_\theta$ are linear in $x_0$ and $x_t$
The model predicts $\hat{x}_0 = (x_t - \sqrt{1-\bar{\alpha}_t}\epsilon_\theta)/\sqrt{\bar{\alpha}_t}$

Papers

Denoising Diffusion Probabilistic Models Improved Denoising Diffusion Probabilistic Models

Blogs

What are Diffusion Models? — Lilian Weng The Annotated Diffusion Model — Hugging Face

Definition

A parameterization is the choice of what the noise-prediction network outputs, whether the noise $\epsilon$ , the clean data $x_0$ , or the velocity $v = \sqrt{\bar{\alpha}_t}\,\epsilon - \sqrt{1 - \bar{\alpha}_t}\,x_0$ . All are algebraically equivalent but differ in gradient conditioning across noise levels; $v$ -prediction interpolates between $\epsilon$ at low noise and $x_0$ at high noise for stable training throughout.

In this topic

1Epsilon-Prediction
2X0-Prediction
3V-Prediction

1 of 3

Epsilon-Prediction

$\hat{x}_0 = \frac{x_t - \sqrt{1 - \bar{\alpha}_t}\, \epsilon_\theta(x_t, t)}{\sqrt{\bar{\alpha}_t}}$

The network predicts the noise $\epsilon$ that was added to the clean data $x_0$ to produce $x_t$ . The clean data is then recovered by rearranging the forward equation. This is the original DDPM parameterization and is simple to implement. At high noise levels ( $\bar{\alpha}_t$ small), $\epsilon$ -prediction works well because the target $\epsilon \sim \mathcal{N}(0, I)$ has constant variance across timesteps. However, at very low noise levels, the $\hat{x}_0$ recovery amplifies errors by dividing by $\sqrt{\bar{\alpha}_t} \approx 0$ .

Mathematical Intuition

In $\epsilon$ -prediction, the target $\epsilon \sim \mathcal{N}(0,I)$ has constant unit variance at all timesteps, providing a well-conditioned regression target. However, recovering $x_0$ requires dividing by $\sqrt{\bar{\alpha}_t}$ : $\hat{x}_0 = (x_t - \sqrt{1-\bar{\alpha}_t}\epsilon_\theta)/\sqrt{\bar{\alpha}_t}$ . At high noise ( $\bar{\alpha}_t \to 0$ ), this division amplifies errors by $1/\sqrt{\bar{\alpha}_t}$ . The effective loss on $x_0$ is $\text{SNR}(t) \cdot \|x_0 - \hat{x}_0\|^2$ , naturally upweighting low-noise steps.

Example:

If the noise predictor has MSE error $\delta$ at $\bar{\alpha}_t = 0.01$ , what is the error in the $x_0$ prediction?

2 of 3

X0-Prediction

$\hat{\epsilon} = \frac{x_t - \sqrt{\bar{\alpha}_t}\, x_{0,\theta}(x_t, t)}{\sqrt{1 - \bar{\alpha}_t}}$

Instead of predicting noise, the network directly predicts the clean data $x_0$ . This is naturally better at low noise levels where $x_t \approx x_0$ , but problematic at high noise where the clean data prediction must be extracted from nearly pure noise---a much harder regression target with high variance. The loss becomes $\|x_0 - x_{0,\theta}(x_t, t)\|^2$ . Ramesh et al. (DALL-E 2) and some image super-resolution models use $x_0$ -prediction for its stability at low noise.

Mathematical Intuition

In $x_0$ -prediction, the target is the clean data itself. The loss $\|x_0 - x_{0,\theta}(x_t,t)\|^2$ has constant magnitude with respect to the target, but the difficulty of predicting $x_0$ from $x_t$ varies dramatically: at low noise, $x_t \approx x_0$ (trivial); at high noise, $x_t \approx \epsilon$ (must hallucinate $x_0$ from noise). Converting to $\epsilon$ -prediction loss: $\|\hat{\epsilon} - \epsilon\|^2 = \text{SNR}(t)^{-1}\|x_0 - \hat{x}_0\|^2$ , so $x_0$ -prediction implicitly upweights high-noise steps.

Example:

At $\bar{\alpha}_t = 0.99$ (low noise), compare the difficulty of predicting $\epsilon$ vs $x_0$ .

3 of 3

V-Prediction

$v = \sqrt{\bar{\alpha}_t}\, \epsilon - \sqrt{1 - \bar{\alpha}_t}\, x_0$

Salimans & Ho (2022) introduced $v$ -prediction, where the target is a velocity-like quantity that smoothly interpolates between $\epsilon$ at high noise and $-x_0$ at low noise. At $\bar{\alpha}_t \approx 1$ (low noise): $v \approx \epsilon$ (behaves like $\epsilon$ -prediction). At $\bar{\alpha}_t \approx 0$ (high noise): $v \approx -x_0$ (behaves like $x_0$ -prediction). This provides the best of both worlds: stable gradients at all noise levels. Recovery: $\hat{x}_0 = \sqrt{\bar{\alpha}_t}\, x_t - \sqrt{1-\bar{\alpha}_t}\, v_\theta$ and $\hat{\epsilon} = \sqrt{1-\bar{\alpha}_t}\, x_t + \sqrt{\bar{\alpha}_t}\, v_\theta$ .

Mathematical Intuition

The velocity $v = \sqrt{\bar{\alpha}_t}\epsilon - \sqrt{1-\bar{\alpha}_t}x_0$ is a rotation of the $(\epsilon, x_0)$ pair by angle $\arctan(\sqrt{(1-\bar{\alpha}_t)/\bar{\alpha}_t})$ . The $v$ -prediction loss decomposes as $\mathbb{E}[\|v-v_\theta\|^2] = \bar{\alpha}_t\|\epsilon-\hat{\epsilon}\|^2 + (1-\bar{\alpha}_t)\|x_0-\hat{x}_0\|^2$ , automatically interpolating between $\epsilon$ - and $x_0$ -prediction with weights that match the signal-noise balance at each $t$ . This provides uniform effective gradient magnitude across all noise levels.

Example:

Verify that $v$ -prediction reduces to $\epsilon$ -prediction at $\bar{\alpha}_t = 1$ and $x_0$ -prediction at $\bar{\alpha}_t = 0$ .

Theory Exercise

Problem:

Show that the $v$ -prediction loss $\|v - v_\theta(x_t, t)\|^2$ is equivalent to a weighted combination of $\epsilon$ -prediction and $x_0$ -prediction losses. Specifically, prove that $\|v - v_\theta\|^2 = \bar{\alpha}_t \|\epsilon - \hat{\epsilon}\|^2 + (1-\bar{\alpha}_t) \|x_0 - \hat{x}_0\|^2$ .

Hints:

Write $v = \sqrt{\bar{\alpha}_t}\epsilon - \sqrt{1-\bar{\alpha}_t}x_0$ and similarly for $v_\theta$
Compute the difference $v - v_\theta$ in terms of $\epsilon - \hat{\epsilon}$ and $x_0 - \hat{x}_0$
Use the fact that for optimal predictions, $\epsilon$ and $x_0$ errors are orthogonal in expectation

Papers

Progressive Distillation for Fast Sampling of Diffusion Models Denoising Diffusion Probabilistic Models

Blogs

What are Diffusion Models? — Lilian Weng Generative Modeling by Estimating Gradients of the Data Distribution — Yang Song

Definition

A loss weighting strategy assigns a per-timestep weight $w(t)$ to the noise-prediction loss, controlling how training effort is distributed across noise levels. Choices range from uniform weighting through SNR-based weighting to min-SNR clipping and perceptual weighting, each trading likelihood against sample quality by emphasizing different regions of the signal-to-noise spectrum.

In this topic

1SNR-Based Weighting
2Min-SNR Loss Weighting
3Perceptual Loss Weighting

1 of 3

SNR-Based Weighting

$\text{SNR}(t) = \frac{\bar{\alpha}_t}{1 - \bar{\alpha}_t}, \quad w(t) = \text{SNR}(t)$

The signal-to-noise ratio at timestep $t$ measures the relative power of signal to noise in $x_t$ . SNR-based weighting multiplies the loss by $\text{SNR}(t)$ , which gives higher weight to low-noise timesteps (where SNR is high) and lower weight to high-noise timesteps. This is equivalent to the VLB weighting and optimizes the likelihood bound. However, it under-weights high-noise timesteps where the model learns global composition, often producing sharper but less coherent samples.

Mathematical Intuition

SNR weighting $w(t) = \text{SNR}(t)$ corresponds to the true VLB objective when used with $x_0$ -prediction. In $\epsilon$ -prediction, the simple loss already implicitly applies SNR weighting to the $x_0$ reconstruction error. The dynamic range of SNR spans $10^4$ -- $10^6$ across timesteps, causing extreme gradient imbalance: the first 10 timesteps can contribute more gradient than the remaining 990 combined. This is why pure SNR weighting underperforms despite being theoretically motivated.

Example:

Compare SNR at $t = 100$ and $t = 900$ for the linear DDPM schedule.

2 of 3

Min-SNR Loss Weighting

$w(t) = \min\left(\text{SNR}(t),\, \gamma\right)$

Hang et al. (2023) proposed Min-SNR- $\gamma$ weighting, which clips the SNR weight at a maximum value $\gamma$ (typically $\gamma = 5$ ). This prevents the loss from being dominated by the easiest timesteps (very low noise, very high SNR) while still giving some preference to perceptually important noise levels. Min-SNR-5 consistently outperforms both uniform and pure SNR weighting on FID and IS metrics across different datasets and architectures. It achieves this by capping the maximum attention given to any single noise level.

Mathematical Intuition

Min-SNR- $\gamma$ truncates the weighting: $w(t) = \min(\text{SNR}(t), \gamma)$ . For $\gamma = 5$ : timesteps with $\bar{\alpha}_t > 5/6$ get weight 5 (capped), while timesteps with $\bar{\alpha}_t < 5/6$ get weight $\text{SNR}(t) < 5$ (uncapped). The gradient contribution becomes $\sum_t w(t)\|\epsilon - \epsilon_\theta\|^2$ , which is bounded above by $\gamma$ per timestep. This eliminates the pathological concentration of gradient on near-clean timesteps while preserving the relative ordering for high-noise steps.

Example:

With $\gamma = 5$ , what fraction of timesteps have their weight clipped (for DDPM linear schedule)?

3 of 3

Perceptual Loss Weighting

$w(t) \propto \frac{d\, \text{SNR}(t)}{dt} \cdot \text{LPIPS}(x_0, \hat{x}_0(t))$

An alternative approach weights timesteps by their perceptual importance, measured by how much perceptual quality (e.g., LPIPS) changes when the model improves at that timestep. Choi et al. (2022) showed that mid-noise timesteps ( $t \approx 200$ -- $600$ in DDPM) contribute most to perceptual quality: they determine textures, edges, and local structure. Very low noise (fine grain) and very high noise (global layout) contribute less to human visual perception. This motivates weighting schemes that emphasize the perceptually critical mid-range.

Mathematical Intuition

Perceptual importance peaks at intermediate noise levels ( $t/T \approx 0.3$ -- $0.6$ ) because this is where the model determines textures, edges, and mid-frequency structure---features that dominate human visual perception. Very low noise encodes imperceptible high-frequency details (PSNR improves but LPIPS is flat), while very high noise encodes class-level semantics (a face vs a landscape) that are already determined by the coarsest features. Optimal perceptual weighting concentrates capacity in the sweet spot where LPIPS improvement per gradient step is maximized.

Example:

Why are mid-noise timesteps the most perceptually important?

Theory Exercise

Problem:

Derive the relationship between the DDPM VLB weight $w_t^{\text{VLB}}$ and the SNR. Show that $w_t^{\text{VLB}} \propto \text{SNR}(t-1) - \text{SNR}(t)$ (the change in SNR between adjacent timesteps).

Hints:

Start from $w_t^{VLB} = \beta_t^2 / (2\tilde{\beta}_t \alpha_t (1-\bar{\alpha}_t))$
Express $\tilde{\beta}_t$ in terms of $\bar{\alpha}_t$ and $\bar{\alpha}_{t-1}$
Use $\text{SNR}(t) = \bar{\alpha}_t / (1 - \bar{\alpha}_t)$

Papers

Efficient Diffusion Training via Min-SNR Weighting Strategy Improved Denoising Diffusion Probabilistic Models

Blogs

What are Diffusion Models? — Lilian Weng Generative Modeling by Estimating Gradients of the Data Distribution — Yang Song

Score Functions

Denoising Score Matching

Reverse SDE

DDPM Objective

Noise Predictor

Loss Weighting

6 topics in this chapter

Score Functions

Definition

In this topic

Definition and Geometric Meaning

Score of a Gaussian Mixture

Score for Noisy Distributions

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank

Denoising Score Matching

Definition

In this topic

The Score Matching Objective

Denoising Score Matching Theorem

From Score Matching to Noise Prediction

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank

The Reverse SDE

Definition

In this topic

Anderson's Reverse-Time SDE

The Probability Flow ODE

Discretizing the Reverse Process

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank

DDPM Training Objective

Definition

In this topic

The Variational Lower Bound

Simplified Training Loss

VLB vs Simple Loss in Practice

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank

Parameterizing the Noise Predictor

Definition

In this topic

Epsilon-Prediction

X0-Prediction

V-Prediction

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank

Loss Weighting Strategies

Definition

In this topic

SNR-Based Weighting

Min-SNR Loss Weighting

Perceptual Loss Weighting

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank