Week 3

Chapter 3: Forward Diffusion Process

The forward diffusion process is the foundation of all diffusion models. It defines how clean data is systematically corrupted by adding Gaussian noise over a sequence of timesteps, transforming any data distribution into pure noise. Understanding this process---noise schedules, reparameterization, and the continuous-time formulation---is essential for training and sampling.

Chapter Overview

The forward diffusion process is conceptually simple but mathematically rich. Starting from a clean data sample $x_0$ , we progressively add small amounts of Gaussian noise over $T$ timesteps, producing a sequence $x_1, x_2, \ldots, x_T$ where $x_T$ is nearly indistinguishable from pure Gaussian noise. This defines a fixed (non-learned) Markov chain that gradually destroys structure in the data.

The elegance of diffusion models lies in the fact that this destruction process is easy to define and analyze, while the generative model learns to reverse it. The forward process gives us: (1) a training procedure---we can sample any noisy version $x_t$ directly from $x_0$ without simulating the full chain, and (2) a starting point for generation---we begin from pure noise and iteratively denoise.

Two mathematical frameworks describe this process. The discrete-time formulation (DDPM) defines $T$ discrete steps with a variance schedule $\beta_1, \ldots, \beta_T$ . The continuous-time formulation (Score SDE) uses stochastic differential equations, treating time as continuous $t \in [0, 1]$ . Both frameworks are equivalent in the limit and each offers distinct advantages for analysis and implementation.

This chapter covers:

Adding Noise to Data: The transition kernel $q(x_t | x_{t-1})$ and visualization of progressive corruption
Noise Schedules: How the variance schedule $\beta_t$ controls the rate of noise injection
The Forward SDE: Continuous-time formulation with VP-SDE and VE-SDE
Reparameterization Trick: Sampling $x_t$ directly from $x_0$ in a single step
Signal-to-Noise Ratio: Quantifying information content at each timestep
Continuous vs Discrete Time: Connecting the two frameworks and practical trade-offs

Chapter Roadmap

Click any topic to jump in

Adding Noise

Gradually corrupting data with Gaussian noise until it becomes pure noise.

The Forward Transition KernelThe Forward Markov ChainVisualizing Progressive Corruption

How to add noise

Schedules and shortcuts

Noise Schedules

Linear, cosine, and learned schedules — controlling how fast information is destroyed.

Linear Schedule (DDPM)Cosine Schedule (Improved DDPM)Learned and Adaptive Schedules

Reparameterization

Sampling any timestep directly from x₀ — making training efficient.

Deriving the Closed-Form DistributionThe Alpha-Bar DerivationEfficient Training with Reparameterization

Unified perspectives

SNR and SDEs

SNR

Signal-to-noise ratio as the unified view of noise schedules.

SNR DefinitionLog-SNR and Its PropertiesMin-SNR Loss Weighting

Forward SDE

The continuous-time formulation — Ornstein-Uhlenbeck and Itô calculus.

General Forward SDEVariance Preserving SDE (VP-SDE)Variance Exploding SDE (VE-SDE)VP-SDE vs VE-SDE Comparison

Bridging frameworks

Continuous vs Discrete

Connecting DDPM discrete steps to the SDE framework.

Discrete-Time Formulation (DDPM)Continuous-Time Formulation (Score SDE)Connecting Discrete and ContinuousPractical Considerations

The forward process defines how data is corrupted step by step. At each timestep $t$ , we take the previous sample $x_{t-1}$ and add a small amount of Gaussian noise controlled by a variance parameter $\beta_t$ . This produces a Markov chain: each step depends only on the immediately preceding state, not on the full history.

Key insight: The noise added at each step is small (typical $\beta_t \in [10^{-4}, 0.02]$ ), so adjacent samples $x_t$ and $x_{t-1}$ look very similar. This is crucial because it means the reverse process (going from $x_t$ back to $x_{t-1}$ ) can be approximated by a Gaussian---a tractable distribution we can learn with a neural network.

In this topic

1The Forward Transition Kernel

2The Forward Markov Chain

3Visualizing Progressive Corruption

1 of 3

The Forward Transition Kernel

$q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1 - \beta_t}\, x_{t-1},\; \beta_t \mathbf{I})$

Each step of the forward process is a Gaussian perturbation. The mean is $\sqrt{1 - \beta_t}\, x_{t-1}$ , which slightly shrinks the signal, and the variance is $\beta_t \mathbf{I}$ , which adds isotropic noise. The scaling factor $\sqrt{1 - \beta_t}$ ensures that the variance of $x_t$ does not explode over time. Together, shrinking the signal and adding noise gradually transforms any data distribution into a standard Gaussian $\mathcal{N}(0, \mathbf{I})$ as $t \to T$ .

Mathematical Intuition

The transition kernel $q(x_t | x_{t-1}) = \mathcal{N}(\sqrt{1-\beta_t}\, x_{t-1}, \beta_t I)$ is a contraction mapping in distribution space. The scaling $\sqrt{1-\beta_t}$ shrinks the signal variance by a factor $1-\beta_t$ , while the additive noise injects exactly $\beta_t$ of fresh variance. Together these preserve unit total variance: $\text{Var}(x_t) = (1-\beta_t)\text{Var}(x_{t-1}) + \beta_t = 1$ whenever $\text{Var}(x_{t-1})=1$ . This variance-preserving property is what makes the process converge to $\mathcal{N}(0,I)$ rather than diverging.

Example:

If $\beta_t = 0.01$ and $x_{t-1} = [3.0, -1.0]$ , what is the distribution of $x_t$ ?

2 of 3

The Forward Markov Chain

$q(x_{1:T} | x_0) = \prod_{t=1}^{T} q(x_t | x_{t-1})$

The full forward process is a Markov chain: the joint distribution of all noisy samples $x_1, \ldots, x_T$ given $x_0$ factorizes as a product of transition kernels. The Markov property means $x_t$ is conditionally independent of $x_0, \ldots, x_{t-2}$ given $x_{t-1}$ . This factorization is what makes the process tractable---we only need to specify one-step transitions, and the entire chain is determined.

Mathematical Intuition

The Markov factorization $q(x_{1:T}|x_0) = \prod_t q(x_t|x_{t-1})$ means the entire forward process is determined by its one-step transitions. In information-theoretic terms, the data processing inequality guarantees mutual information $I(x_0; x_t)$ is monotonically non-increasing in $t$ . Each step irreversibly destroys information, and the chain of conditional independence constraints is what makes the reverse process decompose into tractable single-step denoising problems.

Example:

Why is the Markov property important for making the reverse process tractable?

3 of 3

Visualizing Progressive Corruption

$x_t = \sqrt{1 - \beta_t}\, x_{t-1} + \sqrt{\beta_t}\, \epsilon, \quad \epsilon \sim \mathcal{N}(0, \mathbf{I})$

Starting from a clean image $x_0$ , the forward process generates samples at increasing noise levels. At early timesteps ( $t \approx 0$ ), the image is nearly perfect. At intermediate timesteps ( $t \approx T/2$ ), large-scale structure (shapes, colors) is still visible but details are lost. At late timesteps ( $t \approx T$ ), the sample is indistinguishable from random noise. The rate at which structure is destroyed depends critically on the noise schedule $\{\beta_t\}$ .

Mathematical Intuition

At timestep $t$ , the noisy sample is $x_t = \sqrt{\bar{\alpha}_t}\,x_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon$ . The signal-to-noise ratio $\text{SNR}(t) = \bar{\alpha}_t/(1-\bar{\alpha}_t)$ determines what information survives. High-frequency details (edges, textures) are destroyed first because they occupy a smaller fraction of the total signal energy. Low-frequency structure (overall shape, color) persists until $\bar{\alpha}_t \ll 1$ . This multi-scale destruction is why diffusion models naturally generate coarse-to-fine.

Example:

For a 256x256 image with $T = 1000$ and linear schedule $\beta_1 = 10^{-4}$ , $\beta_T = 0.02$ , at what timestep is roughly half the signal energy destroyed?

Theory Exercise

Problem:

Prove that if $q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1 - \beta_t}\, x_{t-1}, \beta_t \mathbf{I})$ and we start from data with $\text{Var}(x_0) = 1$ , then $\text{Var}(x_t) = 1$ for all $t$ (the variance is preserved at each step).

Hints:

Use the law of total variance or compute Var(x_t) directly from the transition
Recall that for $x_t = ax_{t-1} + b\epsilon$, we have $\text{Var}(x_t) = a^2 \text{Var}(x_{t-1}) + b^2$
Substitute $a = \sqrt{1 - \beta_t}$ and $b = \sqrt{\beta_t}$

Papers

Denoising Diffusion Probabilistic Models Score-Based Generative Modeling through Stochastic Differential Equations

Blogs

What are Diffusion Models? — Lilian Weng Generative Modeling by Estimating Gradients of the Data Distribution — Yang Song

Mathematical Foundations

Reverse Process & Denoising

Chapter 3: Forward Diffusion Process

Chapter Overview

Chapter Roadmap

Adding Noise

Noise Schedules

Reparameterization

SNR

Forward SDE

Continuous vs Discrete

Adding Noise to Data

In this topic

The Forward Transition Kernel

The Forward Markov Chain

Visualizing Progressive Corruption

Theory Exercise

Problem:

Hints:

Noise Schedules

The Forward SDE

Reparameterization Trick

Signal-to-Noise Ratio

Continuous vs Discrete Time