📘

Debiased Estimator of the ELBO Weighting

HardDiffusion Models 📚 Diffusion Models 1

Problem Statement

The variational bound decomposes into per-timestep KL terms. Given per-step eps-losses and their SNRs, compute the true ELBO-weighted total loss and compare it to the "simple" (unweighted) DDPM loss, reporting the ratio.

Background

The negative log-likelihood bound weights the eps-loss at each interior timestep by half the drop in SNR:

$w_t = \tfrac{1}{2}\big(\text{SNR}_{t-1} - \text{SNR}_t\big)$

with SNR_{-1} taken as SNR_0 (so the first term has weight 0). The ELBO loss is sum_t w_t * eps_loss_t; the simple loss is sum_t eps_loss_t. Report both and their ratio (ELBO / simple).

Your Task

Implement:

def elbo_vs_simple(eps_loss, snr):

eps_loss[t], snr[t]: aligned per-timestep lists.

Return a dict with "elbo", "simple", "ratio", each rounded to 4 decimals. If simple == 0, ratio is 0.0.

Input Format

eps_loss: list of T non-negative losses.
snr: list of T SNRs (decreasing).

Output Format

A dict of three floats.

Sample

print(elbo_vs_simple([1.0, 1.0, 1.0], [8.0, 4.0, 2.0]))

Output:

{'elbo': 3.0, 'simple': 3.0, 'ratio': 1.0}

Example:

Input:

print(elbo_vs_simple([1.0, 1.0, 1.0], [8.0, 4.0, 2.0]))

Output:

{'elbo': 3.0, 'simple': 3.0, 'ratio': 1.0}

Reasoning:

Initialize the previous SNR to the first value, $8.0$ , to handle the boundary condition where the weight for the first timestep is zero.
At $t=0$ , calculate the weight as $w_0 = \frac{1}{2}(8.0 - 8.0) = 0$ , contributing $0 \times 1.0 = 0$ to the ELBO sum.
At $t=1$ , update the previous SNR to $4.0$ and compute $w_1 = \frac{1}{2}(8.0 - 4.0) = 2.0$ , adding $2.0 \times 1.0 = 2.0$ to the ELBO sum.
At $t=2$ , update the previous SNR to $2.0$ and compute $w_2 = \frac{1}{2}(4.0 - 2.0) = 1.0$ , adding $1.0 \times 1.0 = 1.0$ to the ELBO sum, resulting in a total ELBO of $3.0$ .
Compute the simple loss as the sum of all $\epsilon$ -losses: $1.0 + 1.0 + 1.0 = 3.0$ , and determine the ratio as $3.0 / 3.0 = 1.0$ .
The final output is {'elbo': 3.0, 'simple': 3.0, 'ratio': 1.0}

Constraints:

len(eps_loss) == len(snr) == T, 1 <= T <= 100000.
w_t = 0.5*(SNR_{t-1} - SNR_t) with SNR_{-1} = SNR_0 (first weight 0).
elbo = sum w_t*eps_loss_t, simple = sum eps_loss_t; round all to 4 decimals.

More from Diffusion Models 1

Classifier-Free Guidance: Uncond/Cond MixingEasy DDPM Posterior Mean and Variance from x0Medium Adaptive CFG Scheduling over the TrajectoryHard Classifier Gradient Guidance StepMedium Negative Prompt as the Unconditional BranchMedium

1. Background Knowledge

In Diffusion Probabilistic Models (DDPM), training minimizes a variational lower bound (ELBO) on the negative log-likelihood. The ELBO decomposes into a sum of per-timestep Kullback-Leibler (KL) divergence terms. For the standard Gaussian parameterization, each interior KL term is proportional to the expected squared error between the true noise $\epsilon$ and the model's prediction $\epsilon_\theta$ , i.e., the epsilon-loss $\mathcal{L}_\epsilon(t) = \mathbb{E}\left[\|\epsilon - \epsilon_\theta(x_t, t)\|^2\right]$ .

The crucial insight is that these terms are not equally weighted. The weight for timestep $t$ is determined by the change in the Signal-to-Noise Ratio (SNR) between adjacent timesteps. Specifically, the weight is half the drop in SNR: $w_t = \frac{1}{2}(\text{SNR}_{t-1} - \text{SNR}_t)$ . This weighting arises because the KL divergence between two Gaussians with the same mean but different variances depends on the ratio of their variances, and the SNR is the reciprocal of the noise variance relative to the signal variance.

In practice, the "simple" DDPM loss often used in the original paper is just the unweighted sum (or mean) of the epsilon-losses across all timesteps. While this is a valid objective, it does not exactly match the ELBO. The ratio $\frac{\text{ELBO}}{\text{Simple}}$ quantifies how much the proper weighting changes the total loss. If the SNR drops are uniform, the weights are uniform, and the ratio is 1. If the SNR drops are non-uniform (which is common in practice, e.g., with cosine or linear schedules), the ratio deviates from 1, indicating that the simple loss over- or under-weights certain timesteps.

2. Algorithm Approach

The problem is a straightforward weighted summation task. The core algorithm involves:

Iterating through the timesteps $t = 0$ to $T-1$ .
Computing the weight $w_t$ for each timestep using the SNR values at $t$ and $t-1$ .
Accumulating the weighted epsilon-loss for the ELBO and the unweighted epsilon-loss for the simple loss.
Calculating the ratio of the two sums, handling the edge case where the simple loss is zero.

This is a linear scan problem with no complex data structures required. The key is correctly handling the boundary condition for $t=0$ .

👑

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

🔒

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.