Diffusion Models

Before diffusion models, generative modeling faced a fundamental dilemma:

• GANs (Goodfellow, 2014): Produce sharp images but suffer from mode collapse (generator produces limited diversity), training instability (discriminator/generator oscillation), and no density estimation (can't compute likelihood). Hyperparameter sensitivity makes training unreliable — small changes to learning rate, architecture, or batch size can cause divergence.

• VAEs (Kingma & Welling, 2014): Stable training via ELBO optimization, can compute likelihood, but produce blurry outputs because the MSE reconstruction loss averages over modes. The latent space is forced to be Gaussian, which may not match the true posterior.

• Autoregressive models (PixelCNN, 2016): Exact likelihood, no mode collapse, but generation is sequential — must generate one pixel at a time (millions of forward passes for one image). Cannot be parallelized during generation.

• Flow models (RealNVP, Glow): Exact likelihood via invertible transformations, but architecturally constrained by the invertibility requirement — can't use standard convolutions, attention, or residual connections.

The core question: Can we design a generative model that combines GANs' sample quality with VAEs' training stability and flow models' tractability?

Forward Diffusion defines a Markov chain that gradually adds Gaussian noise:

At each step $t \in \{1, ..., T\}$, add noise controlled by variance $\beta_t$:

$q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t} \, x_{t-1}, \, \beta_t I)$

This means: $x_t = \sqrt{1-\beta_t} \cdot x_{t-1} + \sqrt{\beta_t} \cdot \epsilon_t$ where $\epsilon_t \sim \mathcal{N}(0, I)$.

The coefficient $\sqrt{1-\beta_t}$ slightly shrinks the signal, and $\sqrt{\beta_t}$ adds fresh noise. Over many steps, the signal-to-noise ratio monotonically decreases until the image is indistinguishable from pure Gaussian noise.

The reparameterization trick — direct sampling from $x_0$:

Define $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s$. Then, applying the recursion $t$ times:

$q(x_t | x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} \, x_0, \, (1-\bar{\alpha}_t) I)$

Or equivalently: $x_t = \sqrt{\bar{\alpha}_t} \cdot x_0 + \sqrt{1-\bar{\alpha}_t} \cdot \epsilon$ where $\epsilon \sim \mathcal{N}(0, I)$

This is critical for training efficiency: we can sample ANY noisy version $x_t$ directly from the clean image $x_0$ in one step — no need to iterate through timesteps 1 to $t$. During training, we randomly sample $t \sim \text{Uniform}(1, T)$ for each training example.

Concrete values with the linear schedule ($\beta_1 = 10^{-4}$, $\beta_T = 0.02$, $T = 1000$):

By $t = 500$, the image is roughly half signal, half noise. By $t = 1000$, it is 99.7% noise — statistically indistinguishable from $\mathcal{N}(0, I)$.