Diffusion Models

Denoising Diffusion Probabilistic Models (DDPMs) are generative models that learn to produce images by reversing a gradual noising process. The core idea: define a forward process that progressively destroys data by adding Gaussian noise over $T = 1000$ steps until the image becomes pure noise, then train a neural network to learn the reverse process — denoising one step at a time, gradually transforming noise back into coherent images.

Why diffusion models won: Unlike GANs (adversarial training → mode collapse, training instability) and VAEs (evidence lower bound → blurry outputs), DDPMs optimize a simple mean squared error loss between predicted and actual noise: $L = \mathbb{E}[\|\epsilon - \epsilon_\theta(x_t, t)\|^2]$. This loss is stable, well-behaved, and doesn't require adversarial dynamics or carefully balanced architectures.

Key results from the DDPM paper (Ho et al., 2020):

• FID 3.17 on CIFAR-10 (256×256), surpassing all GANs at the time
• Inception Score 9.46 on CIFAR-10
• Log-likelihood competitive with autoregressive models

DDPMs form the foundation of virtually all modern image generation systems: Stable Diffusion (Latent Diffusion + CLIP conditioning), DALL-E 2 (CLIP + diffusion prior + decoder), Midjourney, Imagen (T5 text encoder + cascaded diffusion), and Sora (video diffusion). The architecture has extended to audio (AudioLDM), 3D (DreamFusion), video (Stable Video Diffusion), and molecular design (DiffDock).