Loading...
The forward diffusion process is the foundation of all diffusion models. It defines how clean data is systematically corrupted by adding Gaussian noise over a sequence of timesteps, transforming any data distribution into pure noise. Understanding this process---noise schedules, reparameterization, and the continuous-time formulation---is essential for training and sampling.
The forward diffusion process is conceptually simple but mathematically rich. Starting from a clean data sample , we progressively add small amounts of Gaussian noise over timesteps, producing a sequence where is nearly indistinguishable from pure Gaussian noise. This defines a fixed (non-learned) Markov chain that gradually destroys structure in the data.
The elegance of diffusion models lies in the fact that this destruction process is easy to define and analyze, while the generative model learns to reverse it. The forward process gives us: (1) a training procedure---we can sample any noisy version directly from without simulating the full chain, and (2) a starting point for generation---we begin from pure noise and iteratively denoise.
Two mathematical frameworks describe this process. The discrete-time formulation (DDPM) defines discrete steps with a variance schedule . The continuous-time formulation (Score SDE) uses stochastic differential equations, treating time as continuous . Both frameworks are equivalent in the limit and each offers distinct advantages for analysis and implementation.
This chapter covers:
Click any topic to jump in
Gradually corrupting data with Gaussian noise until it becomes pure noise.
Schedules and shortcuts
Linear, cosine, and learned schedules — controlling how fast information is destroyed.
Sampling any timestep directly from x₀ — making training efficient.
SNR and SDEs
Signal-to-noise ratio as the unified view of noise schedules.
The continuous-time formulation — Ornstein-Uhlenbeck and Itô calculus.
Connecting DDPM discrete steps to the SDE framework.
The forward process defines how data is corrupted step by step. At each timestep , we take the previous sample and add a small amount of Gaussian noise controlled by a variance parameter . This produces a Markov chain: each step depends only on the immediately preceding state, not on the full history.
Key insight: The noise added at each step is small (typical ), so adjacent samples and look very similar. This is crucial because it means the reverse process (going from back to ) can be approximated by a Gaussian---a tractable distribution we can learn with a neural network.
Each step of the forward process is a Gaussian perturbation. The mean is , which slightly shrinks the signal, and the variance is , which adds isotropic noise. The scaling factor ensures that the variance of does not explode over time. Together, shrinking the signal and adding noise gradually transforms any data distribution into a standard Gaussian as .
The transition kernel is a contraction mapping in distribution space. The scaling shrinks the signal variance by a factor , while the additive noise injects exactly of fresh variance. Together these preserve unit total variance: whenever . This variance-preserving property is what makes the process converge to rather than diverging.
If and , what is the distribution of ?
The full forward process is a Markov chain: the joint distribution of all noisy samples given factorizes as a product of transition kernels. The Markov property means is conditionally independent of given . This factorization is what makes the process tractable---we only need to specify one-step transitions, and the entire chain is determined.
The Markov factorization means the entire forward process is determined by its one-step transitions. In information-theoretic terms, the data processing inequality guarantees mutual information is monotonically non-increasing in . Each step irreversibly destroys information, and the chain of conditional independence constraints is what makes the reverse process decompose into tractable single-step denoising problems.
Why is the Markov property important for making the reverse process tractable?
Starting from a clean image , the forward process generates samples at increasing noise levels. At early timesteps (), the image is nearly perfect. At intermediate timesteps (), large-scale structure (shapes, colors) is still visible but details are lost. At late timesteps (), the sample is indistinguishable from random noise. The rate at which structure is destroyed depends critically on the noise schedule .
At timestep , the noisy sample is . The signal-to-noise ratio determines what information survives. High-frequency details (edges, textures) are destroyed first because they occupy a smaller fraction of the total signal energy. Low-frequency structure (overall shape, color) persists until . This multi-scale destruction is why diffusion models naturally generate coarse-to-fine.
For a 256x256 image with and linear schedule , , at what timestep is roughly half the signal energy destroyed?
Prove that if and we start from data with , then for all (the variance is preserved at each step).