Chapter 3: Forward Diffusion Process

The original DDPM paper (Ho et al., 2020) uses a linear schedule with $\beta_1 = 10^{-4}$ and $\beta_T = 0.02$ over $T = 1000$ steps. This causes $\bar{\alpha}_t$ to decay roughly exponentially, meaning most of the signal destruction happens early: $\bar{\alpha}_t$ is already below 0.5 by $t \approx 260$ and about 0.08 at $t = 500$. The linear schedule is simple and works well but is suboptimal: it spends too many timesteps on nearly pure noise (where almost no signal is left to recover) and too few on the critical mid-noise range.

Nichol & Dhariwal (2021) proposed defining $\bar{\alpha}_t$ directly using a cosine function, then deriving $\beta_t$ from it. The offset $s = 0.008$ prevents $\beta_t$ from being too small near $t = 0$ or too large near $t = T$. The cosine schedule produces a much more uniform rate of information destruction: $\bar{\alpha}_t$ decreases roughly linearly in $t$, meaning the model spends equal capacity on easy, medium, and hard denoising tasks.

Instead of fixing the schedule beforehand, some approaches learn $\beta_t$ jointly with the model or adapt the schedule to the data. Kingma et al. (2021) parameterize the noise schedule as a monotonic neural network. Hoogeboom et al. (2023) show that the optimal schedule depends on data resolution: higher-resolution images need more steps at low noise levels to capture fine details. Practical finding: for most tasks, the cosine schedule is a strong default, and learned schedules provide marginal improvement at the cost of training complexity.

The continuous-time forward process is described by an It^o SDE. Here $f(x, t)$ is the drift coefficient, the deterministic force that pulls the sample toward the origin, while $g(t)$ is the diffusion coefficient setting the noise magnitude, and $dw$ is an infinitesimal Wiener increment, a Gaussian step of variance $dt$. The drift shrinks the signal and the diffusion injects randomness, exactly mirroring the discrete kernel's rescale-and-add-noise pair from the previous topic. Different choices of $f$ and $g$ yield different processes: get their balance wrong and the marginal variance either collapses to zero or explodes without bound, destroying the clean terminal Gaussian we rely on as the starting point for sampling.

The VP-SDE is the continuous-time limit of the DDPM forward process. The drift is $f(x, t) = -\frac{1}{2}\beta(t)x$, which linearly shrinks $x$ toward zero. The diffusion coefficient is $g(t) = \sqrt{\beta(t)}$. The name 'variance preserving' comes from the fact that if the data has unit variance, the process maintains unit variance at all times---the signal lost through the drift is exactly replaced by noise from the diffusion. This corresponds to an Ornstein-Uhlenbeck process with time-varying parameters.

The VE-SDE has zero drift ($f = 0$) and only adds noise, corresponding to the NCSN/SMLD (Song & Ermon, 2019) framework. The data is not scaled down; only increasingly large noise is added. This means the variance of $x_t$ grows (explodes) over time: $\text{Var}(x_t) = \text{Var}(x_0) + \sigma^2(t)$. The noise schedule $\sigma(t)$ is typically geometric, ranging from a small value $\sigma_{\min}$ to a large value $\sigma_{\max}$. The VE-SDE is simpler to analyze (no drift term) but produces samples with higher variance at late timesteps.

VP-SDE (DDPM) scales down the data while adding noise, maintaining unit variance throughout. The terminal distribution is $\mathcal{N}(0, \mathbf{I})$. VE-SDE (SMLD/NCSN) only adds noise without scaling, so the terminal distribution is $\mathcal{N}(0, \sigma_{\max}^2 \mathbf{I})$ with very large variance. In practice, VP-SDE tends to produce better sample quality (the fixed variance simplifies the learning problem), while VE-SDE offers simpler theoretical analysis. Most modern diffusion models use VP-SDE or its close variants.

Because each forward step adds independent Gaussian noise, and sums of independent Gaussians stay Gaussian, we can collapse $t$ sequential steps into one. Starting from $q(x_1 \mid x_0) = \mathcal{N}(\sqrt{\alpha_1}\, x_0, \beta_1 \mathbf{I})$ with $\alpha_t = 1 - \beta_t$, and unrolling the recursion $x_t = \sqrt{\alpha_t}\, x_{t-1} + \sqrt{\beta_t}\, \epsilon_t$, induction gives $x_t = \sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \epsilon$. Here $\bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s$ is the surviving signal fraction and $\epsilon \sim \mathcal{N}(0, \mathbf{I})$. The mean carries the remaining signal; the variance $1 - \bar{\alpha}_t$ is the total accumulated noise built up after all the individual steps combine together. This single closed form is precisely what makes efficient single-step training of diffusion models possible at all.

The cumulative product $\bar{\alpha}_t$ captures the total signal retention after $t$ steps. At $t = 0$, $\bar{\alpha}_0 = 1$ (all signal). As $t$ increases, $\bar{\alpha}_t$ decreases monotonically toward 0. The rate of decrease is determined by the noise schedule. For a linear schedule, $\log \bar{\alpha}_t \approx -\sum_{s=1}^{t} \beta_s$, which grows quadratically, causing $\bar{\alpha}_t$ to decay roughly as a Gaussian function of $t$. Pre-computing $\bar{\alpha}_t$ for all $t$ is essential for efficient training.

The reparameterization trick enables the simple DDPM training algorithm: (1) Sample a data point $x_0$ and a random timestep $t \sim \text{Uniform}(1, T)$. (2) Sample noise $\epsilon \sim \mathcal{N}(0, \mathbf{I})$. (3) Compute $x_t = \sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \epsilon$ in one step. (4) Train the network $\epsilon_\theta$ to predict $\epsilon$ from $x_t$ and $t$. Without the reparameterization, we would need to simulate the full chain for each training sample, making training $T$ times slower.

For the distribution $q(x_t | x_0) = \mathcal{N}(\sqrt{\bar{\alpha}_t}\, x_0, (1 - \bar{\alpha}_t)\mathbf{I})$, the signal power is proportional to $\bar{\alpha}_t$ (the squared coefficient on $x_0$) and the noise power is $1 - \bar{\alpha}_t$. Their ratio gives the SNR. At $t = 0$: $\text{SNR} \to \infty$ (pure signal). At $t = T$: $\text{SNR} \to 0$ (pure noise). The SNR decreases monotonically, and the forward process can be thought of as a journey from infinite SNR to zero SNR.

The log-SNR $\lambda(t)$ is more convenient because: (1) it converts the multiplicative SNR decay into an additive decrease, (2) it has a natural range from $+\infty$ to $-\infty$, (3) $\lambda = 0$ corresponds to the critical point where signal and noise power are equal ($\bar{\alpha}_t = 0.5$), and (4) many theoretical results are cleaner when expressed in terms of $\lambda$. The cosine schedule produces a nearly linear log-SNR trajectory, which is one reason it works well.

Hang et al. (2023) proposed min-SNR-$\gamma$ weighting to address the problem that uniform timestep sampling overweights high-noise timesteps (low SNR). The standard DDPM loss implicitly weights each timestep by $\text{SNR}(t)^{-1}$, which becomes very large for high-noise steps, causing training instability and slow convergence. Min-SNR clips the weight to a maximum of $\gamma$ (typically $\gamma = 5$), preventing high-noise timesteps from dominating the gradient. This simple modification significantly improves sample quality and training speed.

The DDPM formulation uses integer timesteps $1$ through $T$, typically $T = 1000$. The schedule $\{\beta_t\}$ defines $\alpha_t = 1 - \beta_t$, and the network $\epsilon_\theta(x_t, t)$ receives a discrete timestep embedding. Each step $x_t = \sqrt{\alpha_t}\, x_{t-1} + \sqrt{1 - \alpha_t}\, \epsilon_t$ is exactly the reparameterized kernel from earlier topics, now indexed by an integer clock. Its strengths are simplicity and well-understood training dynamics. Its weakness is rigidity: $T$ is fixed before training and governs both sampling cost and the Gaussian-reverse approximation. Too small a $T$ forces large noise jumps that violate that Gaussian approximation and wreck sample quality.

The continuous formulation treats time as a real number. The forward process is an SDE, and the reverse process is also an SDE that depends on the score function $\nabla_x \log p_t(x)$. Advantages: (1) sampling is decoupled from training---you can use any ODE/SDE solver with any number of steps, (2) theoretical analysis is cleaner using tools from stochastic calculus, (3) the framework unifies DDPM (VP-SDE) and NCSN (VE-SDE). The model learns a continuous score function $s_\theta(x, t)$ for all $t \in [0, 1]$.

The discrete schedule $\{\beta_1, \ldots, \beta_T\}$ and continuous schedule $\beta(t)$ are related by treating each discrete step as an interval of width $1/T$. The continuous $\beta(t)$ at time $t$ equals $T \cdot \beta_{\lfloor tT \rfloor + 1}$ (the discrete beta scaled by $T$). As $T \to \infty$, the discrete chain converges to the SDE solution. The cumulative signal retention is $\bar{\alpha}(t) = e^{-\int_0^t \beta(s) ds}$ in continuous time, which equals $\prod_{s=1}^{\lfloor tT \rfloor}(1 - \beta_s)$ in discrete time (up to $O(1/T)$ error).

In practice, models are always trained on discrete timesteps (hardware requires discrete computation). The continuous formulation matters for: (1) deriving better samplers (DDIM, DPM-Solver are based on the continuous ODE), (2) theoretical analysis (proving convergence, bounding errors), (3) interpolating between timesteps during sampling (using fewer steps than $T$). Modern best practice: train with large $T$ (or continuous $t$ uniformly sampled from $[0, 1]$), then use adaptive ODE solvers for sampling that automatically choose step sizes based on local error estimates.