Debiased Estimator of the ELBO Weighting

hardPROSignal-to-Noise Ratio and Loss WeightingDiffusion ModelsDiffusion Models 1

Problem Statement

The variational bound decomposes into per-timestep KL terms. Given per-step eps-losses and their SNRs, compute the true ELBO-weighted total loss and compare it to the "simple" (unweighted) DDPM loss, reporting the ratio.

Background

The negative log-likelihood bound weights the eps-loss at each interior timestep by half the drop in SNR:

$w_t = \tfrac{1}{2}\big(\text{SNR}_{t-1} - \text{SNR}_t\big)$

with SNR_{-1} taken as SNR_0 (so the first term has weight 0). The ELBO loss is sum_t w_t * eps_loss_t; the simple loss is sum_t eps_loss_t. Report both and their ratio (ELBO / simple).

Your Task

Implement:

def elbo_vs_simple(eps_loss, snr):

eps_loss[t], snr[t]: aligned per-timestep lists.

Return a dict with "elbo", "simple", "ratio", each rounded to 4 decimals. If simple == 0, ratio is 0.0.

Input Format

eps_loss: list of T non-negative losses.
snr: list of T SNRs (decreasing).

Output Format

A dict of three floats.

Sample

print(elbo_vs_simple([1.0, 1.0, 1.0], [8.0, 4.0, 2.0]))

Output:

{'elbo': 3.0, 'simple': 3.0, 'ratio': 1.0}

Example

Input

print(elbo_vs_simple([1.0, 1.0, 1.0], [8.0, 4.0, 2.0]))

Output

{'elbo': 3.0, 'simple': 3.0, 'ratio': 1.0}

Reasoning

Initialize the previous SNR to the first value, $8.0$ , to handle the boundary condition where the weight for the first timestep is zero.
At $t=0$ , calculate the weight as $w_0 = \frac{1}{2}(8.0 - 8.0) = 0$ , contributing $0 \times 1.0 = 0$ to the ELBO sum.
At $t=1$ , update the previous SNR to $4.0$ and compute $w_1 = \frac{1}{2}(8.0 - 4.0) = 2.0$ , adding $2.0 \times 1.0 = 2.0$ to the ELBO sum.
At $t=2$ , update the previous SNR to $2.0$ and compute $w_2 = \frac{1}{2}(4.0 - 2.0) = 1.0$ , adding $1.0 \times 1.0 = 1.0$ to the ELBO sum, resulting in a total ELBO of $3.0$ .
Compute the simple loss as the sum of all $\epsilon$ -losses: $1.0 + 1.0 + 1.0 = 3.0$ , and determine the ratio as $3.0 / 3.0 = 1.0$ .
The final output is {'elbo': 3.0, 'simple': 3.0, 'ratio': 1.0}

Constraints

len(eps_loss) == len(snr) == T, 1 <= T <= 100000.
w_t = 0.5*(SNR_{t-1} - SNR_t) with SNR_{-1} = SNR_0 (first weight 0).
elbo = sum w_t*eps_loss_t, simple = sum eps_loss_t; round all to 4 decimals.

Learn this in

Introduction to Generative ModelsDiffusion Study Plan Reverse Process & DenoisingDiffusion Study Plan

More from Diffusion Models 1

Classifier-Free Guidance: Uncond/Cond Mixingeasy DDPM Posterior Mean and Variance from x0medium Adaptive CFG Scheduling over the Trajectoryhard Classifier Gradient Guidance Stepmedium Negative Prompt as the Unconditional Branchmedium

Concept primerPREVIEW

1. Background Knowledge

In Diffusion Probabilistic Models (DDPM), training minimizes a variational lower bound (ELBO) on the negative log-likelihood. The ELBO decomposes into a sum of per-timestep Kullback-Leibler (KL) divergence terms. For the standard Gaussian parameterization, each interior KL term is proportional to the expected squared error between the true noise $\epsilon$ and the model's prediction $\epsilon_\theta$ , i.e., the epsilon-loss $\mathcal{L}_\epsilon(t) = \mathbb{E}\left[\|\epsilon - \epsilon_\theta(x_t, t)\|^2\right]$ .

The crucial insight is that these terms are not equally weighted. The weight for timestep $t$ is determined by the change in the Signal-to-Noise Ratio (SNR) between adjacent timesteps. Specifically, the weight is half the drop in SNR: $w_t = \frac{1}{2}(\text{SNR}_{t-1} - \text{SNR}_t)$ . This weighting arises because the KL divergence between two Gaussians with the same mean but different variances depends on the ratio of their variances, and the SNR is the reciprocal of the noise variance relative to the signal variance.

In practice, the "simple" DDPM loss often used in the original paper is just the unweighted sum (or mean) of the epsilon-losses across all timesteps. While this is a valid objective, it does not exactly match the ELBO. The ratio $\frac{\text{ELBO}}{\text{Simple}}$ quantifies how much the proper weighting changes the total loss. If the SNR drops are uniform, the weights are uniform, and the ratio is 1. If the SNR drops are non-uniform (which is common in practice, e.g., with cosine or linear schedules), the ratio deviates from 1, indicating that the simple loss over- or under-weights certain timesteps.

2. Algorithm Approach

The problem is a straightforward weighted summation task. The core algorithm involves:

Iterating through the timesteps $t = 0$ to $T-1$ .
Computing the weight $w_t$ for each timestep using the SNR values at $t$ and $t-1$ .
Accumulating the weighted epsilon-loss for the ELBO and the unweighted epsilon-loss for the simple loss.
Calculating the ratio of the two sums, handling the edge case where the simple loss is zero.

This is a linear scan problem with no complex data structures required. The key is correctly handling the boundary condition for $t=0$ .

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.