Generative2020

Diffusion Models

Denoising Diffusion Probabilistic Models (DDPM)

Jonathan Ho, Ajay Jain, Pieter Abbeel

Paper Overview

Denoising Diffusion Probabilistic Models (DDPMs) are generative models that learn to produce images by reversing a gradual noising process. The core idea: define a forward process that progressively destroys data by adding Gaussian noise over $T = 1000$ steps until the image becomes pure noise, then train a neural network to learn the reverse process — denoising one step at a time, gradually transforming noise back into coherent images.

Why diffusion models won: Unlike GANs (adversarial training → mode collapse, training instability) and VAEs (evidence lower bound → blurry outputs), DDPMs optimize a simple mean squared error loss between predicted and actual noise: $L = \mathbb{E}[\|\epsilon - \epsilon_\theta(x_t, t)\|^2]$ . This loss is stable, well-behaved, and doesn't require adversarial dynamics or carefully balanced architectures.

Key results from the DDPM paper (Ho et al., 2020):

FID 3.17 on CIFAR-10 (256×256), surpassing all GANs at the time
Inception Score 9.46 on CIFAR-10
Log-likelihood competitive with autoregressive models

DDPMs form the foundation of virtually all modern image generation systems: Stable Diffusion (Latent Diffusion + CLIP conditioning), DALL-E 2 (CLIP + diffusion prior + decoder), Midjourney, Imagen (T5 text encoder + cascaded diffusion), and Sora (video diffusion). The architecture has extended to audio (AudioLDM), 3D (DreamFusion), video (Stable Video Diffusion), and molecular design (DiffDock).

Chapter Roadmap

Click any topic to jump in

Forward Diffusion Process

Gaussian noise schedule that lets you jump to any timestep in closed form — $x_t = \sqrt{\bar\alpha_t} x_0 + \sqrt{1-\bar\alpha_t} \epsilon$.

Reverse Denoising Process

Train a noise predictor with MSE; sample by iteratively subtracting predicted noise from $x_T$ to $x_0$.

Noise Schedule

Linear vs cosine $\beta_t$ — the ramp shape controls where training signal is spent, and cosine wins on FID.

Forward/reverse process defined by the schedule

U-Net Architecture for Denoising

U-Net with time-conditioned FiLM layers and self-attention at low resolutions — the workhorse of modern diffusion.

DDIM: Faster Deterministic Sampling

Non-Markovian deterministic sampler — 20× fewer steps and invertible latent interpolation.

Parameterized by a U-Net with fast samplers

Classifier-Free Guidance (CFG)

Extrapolate in the direction that conditioning changes the prediction — no extra classifier needed, just one net.

Latent Diffusion: Efficient High-Resolution

Run diffusion in a VAE latent space ($64\times$ smaller) — 48× less compute, same perceptual quality.

The forward process is a fixed (non-learned) Markov chain that gradually destroys data by adding Gaussian noise over T = 1000 timesteps. The crucial mathematical property is that we can sample any intermediate noisy version directly from the clean image, without iterating through all previous steps.

The Problem

Before diffusion models, generative modeling faced a fundamental dilemma:

GANs (Goodfellow, 2014): Produce sharp images but suffer from mode collapse (generator produces limited diversity), training instability (discriminator/generator oscillation), and no density estimation (can't compute likelihood). Hyperparameter sensitivity makes training unreliable — small changes to learning rate, architecture, or batch size can cause divergence.
VAEs (Kingma & Welling, 2014): Stable training via ELBO optimization, can compute likelihood, but produce blurry outputs because the MSE reconstruction loss averages over modes. The latent space is forced to be Gaussian, which may not match the true posterior.
Autoregressive models (PixelCNN, 2016): Exact likelihood, no mode collapse, but generation is sequential — must generate one pixel at a time (millions of forward passes for one image). Cannot be parallelized during generation.
Flow models (RealNVP, Glow): Exact likelihood via invertible transformations, but architecturally constrained by the invertibility requirement — can't use standard convolutions, attention, or residual connections.

The core question: Can we design a generative model that combines GANs' sample quality with VAEs' training stability and flow models' tractability?

The Solution

Forward Diffusion defines a Markov chain that gradually adds Gaussian noise:

At each step $t \in \{1, ..., T\}$ , add noise controlled by variance $\beta_t$ :

$q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t} \, x_{t-1}, \, \beta_t I)$

This means: $x_t = \sqrt{1-\beta_t} \cdot x_{t-1} + \sqrt{\beta_t} \cdot \epsilon_t$ where $\epsilon_t \sim \mathcal{N}(0, I)$ .

The coefficient $\sqrt{1-\beta_t}$ slightly shrinks the signal, and $\sqrt{\beta_t}$ adds fresh noise. Over many steps, the signal-to-noise ratio monotonically decreases until the image is indistinguishable from pure Gaussian noise.

The reparameterization trick — direct sampling from $x_0$ :

Define $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s$ . Then, applying the recursion $t$ times:

$q(x_t | x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} \, x_0, \, (1-\bar{\alpha}_t) I)$

Or equivalently: $x_t = \sqrt{\bar{\alpha}_t} \cdot x_0 + \sqrt{1-\bar{\alpha}_t} \cdot \epsilon$ where $\epsilon \sim \mathcal{N}(0, I)$

This is critical for training efficiency: we can sample ANY noisy version $x_t$ directly from the clean image $x_0$ in one step — no need to iterate through timesteps 1 to $t$ . During training, we randomly sample $t \sim \text{Uniform}(1, T)$ for each training example.

Concrete values with the linear schedule ( $\beta_1 = 10^{-4}$ , $\beta_T = 0.02$ , $T = 1000$ ):

Timestep $t$	$\bar{\alpha}_t$	Signal scale $\sqrt{\bar{\alpha}_t}$	Noise scale $\sqrt{1-\bar{\alpha}_t}$	SNR (dB)
0 (clean)	1.0	1.0	0.0	∞
100	0.980	0.990	0.141	16.9
250	0.935	0.967	0.255	11.6
500	0.735	0.857	0.515	4.4
750	0.385	0.620	0.785	-2.1
1000	0.006	0.079	0.997	-22.0

By $t = 500$ , the image is roughly half signal, half noise. By $t = 1000$ , it is 99.7% noise — statistically indistinguishable from $\mathcal{N}(0, I)$ .

Key Points

The forward process is fixed (no learnable parameters) — it's a predetermined Markov chain that adds noise according to a schedule $\{\beta_t\}$

Reparameterization: $x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1-\bar{\alpha}_t} \epsilon$ allows sampling any timestep directly from $x_0$ in one step — critical for training efficiency

The signal-to-noise ratio $\text{SNR}(t) = \bar{\alpha}_t / (1-\bar{\alpha}_t)$ monotonically decreases from $\infty$ to $\approx 0$ over $T = 1000$ steps

With the linear schedule ( $\beta_1 = 10^{-4}$ , $\beta_T = 0.02$ ), the image is half signal/half noise at $t \approx 500$ and 99.7% noise at $t = 1000$

Each step slightly shrinks the signal ( $\times \sqrt{1-\beta_t}$ ) while adding fresh noise ( $+ \sqrt{\beta_t} \epsilon$ ) — the shrinkage ensures the process converges to $\mathcal{N}(0, I)$ regardless of the starting distribution

Mathematical Formulation

Single Step Forward

$x_t = \sqrt{1-\beta_t} x_{t-1} + \sqrt{\beta_t} \epsilon$

Each step mixes the previous sample with fresh Gaussian noise

Direct Sampling

$x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1-\bar{\alpha}_t} \epsilon$

Jump directly to any timestep (used during training)

Mathematical Intuition

The forward process gradually corrupts data: $q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t}\,x_{t-1}, \beta_t I)$ . Because Gaussians compose, you can sample $x_t$ directly from $x_0$ : $x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon$ where $\bar\alpha_t = \prod_{s=1}^{t}(1-\beta_s)$ . As $t \to T$ , $\bar\alpha_t \to 0$ and $x_T \to \mathcal{N}(0, I)$ — pure noise regardless of the starting image.

Premium Content

Upgrade to PixelBank Premium to unlock this content.

Back to All Papers

Study Plans

Diffusion Study Plan 3D CV Study Plan All Papers Practice Problems