Chapter 1: Introduction to Generative Models

GANs (Goodfellow et al., 2014) train two networks in a minimax game: a generator $G$ maps random noise $z \sim p(z)$ to fake samples $G(z)$, and a discriminator $D$ outputs the probability that its input is real. The objective rewards $D$ for driving $D(x)$ toward 1 on real data and $D(G(z))$ toward 0 on fakes, while $G$ is trained to fool it. At the game's equilibrium, $G$'s samples are indistinguishable from real data. GANs yield sharp, high-quality images but train unstably --- mode collapse and vanishing gradients are common --- and provide no way to evaluate the likelihood of a sample.

VAEs (Kingma & Welling, 2014) learn a latent representation $z$ of the data. An encoder maps data to a distribution over latent codes $q(z|x)$, and a decoder reconstructs data from latent codes $P(x|z)$. Training maximizes the Evidence Lower Bound (ELBO), which balances reconstruction quality against keeping the latent space well-structured. VAEs provide stable training and tractable likelihood bounds but produce blurry samples because the decoder must average over posterior uncertainty.

Normalizing flows (Rezende & Mohamed, 2015) transform a simple base distribution (e.g., Gaussian) into a complex data distribution through a sequence of invertible transformations $f_1, f_2, \ldots, f_K$. Because each transformation is invertible with a tractable Jacobian determinant, exact likelihood computation is possible via the change of variables formula. The trade-off is architectural constraints: every layer must be invertible, limiting expressiveness and requiring specialized architectures like RealNVP, Glow, or Neural Spline Flows.

Autoregressive models (van den Oord et al., 2016) decompose the joint distribution into a product of conditionals using the chain rule of probability. Each dimension is generated sequentially, conditioned on all previously generated dimensions. This provides exact, tractable likelihood computation and avoids the blurriness of VAEs. However, generation is inherently sequential --- each dimension must wait for all previous ones --- making sampling slow. Examples include PixelCNN for images and GPT for text.

MLE finds the parameters $\theta$ that maximize the probability the model assigns to the observed data. Here $P_\theta(x_i)$ is the model's density at data point $x_i$ and the sum runs over all $N$ examples; taking the log turns the product of per-example probabilities into a numerically stable sum. Maximizing this is equivalent to minimizing the forward KL divergence $D_{\text{KL}}(P_{\text{data}} \| P_\theta)$ between the true and model distributions. MLE is the gold standard: a model at the global optimum has matched the data distribution exactly. Its weakness is that it needs a computable $P_\theta(x)$ --- the very thing latent-variable models lack.

A likelihood is tractable when $P_\theta(x)$ can be computed exactly in polynomial time. Normalizing flows and autoregressive models have tractable likelihoods by construction. A likelihood is intractable when it requires integrating over latent variables $z$, and this integral has no closed-form solution. VAEs and diffusion models have intractable likelihoods because the marginalization over $z$ involves summing over all possible latent configurations --- an exponentially large or continuous space. For intractable models, we resort to variational bounds.

When the true likelihood is intractable, we optimize a lower bound instead. The ELBO consists of two terms: a reconstruction term that measures how well the decoder $P(x|z)$ reconstructs data from latent codes, and a KL regularization term that penalizes the encoder $q(z|x)$ for deviating from the prior $P(z)$. The gap between $\log P(x)$ and the ELBO equals $D_{\text{KL}}(q(z|x) \| P(z|x))$, which measures how well the approximate posterior matches the true posterior. A tighter ELBO means better training.

In a latent variable model, data generation is a two-step process: (1) sample a latent code $z$ from a simple prior distribution $P(z)$, typically a standard Gaussian, and (2) transform it through a complex decoder $P(x|z)$ to produce observable data. The prior $P(z)$ is deliberately simple --- the complexity is pushed entirely into the decoder. The marginal likelihood $P(x)$ integrates over all possible latent codes, which is why it is typically intractable.

Given observed data $x$, the posterior $P(z \mid x)$ tells us which latent codes could have produced it. By Bayes' rule it equals $P(x \mid z)P(z)$ divided by the marginal $P(x) = \int P(x \mid z')P(z')\,dz'$ --- and that denominator is the same intractable integral from the likelihood topic. This intractability is the fundamental bottleneck of latent-variable models: without $P(x)$ you cannot normalize the posterior. The fix is approximate inference --- replace the true posterior with a tractable family $q(z \mid x)$ chosen to be close in KL divergence. A VAE's encoder network outputs exactly this approximation, amortizing inference across the whole dataset.

The encoder-decoder framework solves the inference problem by training two networks jointly: an encoder $q_\phi(z|x)$ that approximates the intractable posterior (mapping data to latent codes), and a decoder $P_\theta(x|z)$ that maps latent codes back to data. The encoder is parameterized as a Gaussian $q_\phi(z|x) = \mathcal{N}(\mu_\phi(x), \sigma_\phi^2(x))$, where neural networks output the mean and variance. Training optimizes the ELBO, which encourages the encoder to match the true posterior while the decoder learns to reconstruct accurately.

FID (Heusel et al., 2017) is the most widely used metric for image generation. It extracts features from both real and generated images using a pretrained Inception network, then computes the Frechet distance between the two feature distributions (modeled as Gaussians with means $\mu$ and covariances $\Sigma$). Lower FID means the generated distribution is closer to the real distribution. FID captures both quality (are features realistic?) and diversity (does the feature distribution match?), but it assumes Gaussian feature distributions and is sensitive to sample size.

The Inception Score (Salimans et al., 2016) measures two properties using a pretrained Inception classifier: (1) quality --- each generated image should be confidently classified (low entropy $P(y|x)$), and (2) diversity --- the marginal class distribution $P(y)$ should be uniform across all classes (high entropy). The KL divergence between these captures both. Higher IS indicates better quality and diversity. IS has notable limitations: it ignores intra-class diversity, does not compare to real data, and is biased toward ImageNet classes.

Precision and recall (Sajjadi et al., 2018; Kynkaanniemi et al., 2019) separate quality from diversity. Precision measures the fraction of generated samples that fall within the real data distribution (quality). Recall measures the fraction of real data that falls within the generated distribution (diversity/coverage). A model with high precision but low recall generates realistic but repetitive samples (mode collapse). A model with high recall but low precision covers all modes but with poor sample quality. FID conflates both into a single number; precision/recall gives a more nuanced picture.

There is a fundamental tension in generative model evaluation between perceptual quality (does each sample look good to humans?) and distribution matching (does the model capture the true data distribution?). FID and IS use pretrained classifiers as proxies for human judgment, but they can be fooled by adversarial examples or textures that activate classifier features without looking realistic to humans. Human evaluation remains the gold standard for perceptual quality but is expensive and subjective. Recent metrics like CMMD (CLIP Maximum Mean Discrepancy) and DINOv2-based FID try to use better feature extractors, but no single metric fully captures generative quality.

The forward process $q$ gradually adds Gaussian noise to data over $T$ steps (typically $T = 1000$), transforming any data distribution into pure noise. The reverse process $p_\theta$ learns to undo each step of this noising. Because each step makes only a small change, the reverse process at each step is approximately Gaussian --- a simple distribution that neural networks can model accurately. This is the key insight: instead of learning to generate data in one shot (which is very hard), diffusion models break generation into many small, easy denoising steps.

Diffusion models solve GANs' three biggest problems:

1. Training stability: No adversarial game. The training objective is a simple denoising loss (mean squared error), which is stable and well-understood. There is no discriminator to balance against the generator.

2. Mode coverage: The forward process visits all data modes uniformly, and the denoising objective ensures the reverse process covers them all. There is no mechanism for mode collapse because the model is trained on every data point at every noise level.

3. Evaluation: Diffusion models provide tractable likelihood bounds (via the variational lower bound on the reverse process), unlike GANs which have no density estimation capability.

The trade-off is sampling speed: GANs generate in one forward pass, while diffusion models require many iterative steps.

Diffusion models also address VAEs' key weaknesses:

1. Sample sharpness: VAEs produce blurry samples because the decoder averages over posterior uncertainty. Diffusion models generate sharp samples because each denoising step makes a definite prediction, and the iterative process allows the model to commit to specific details progressively.

2. Expressiveness: VAE posteriors are typically simple Gaussians. Diffusion models implicitly define a highly expressive posterior over denoising trajectories, with each step refining the approximation.

3. Architecture flexibility: The denoising network can be any architecture (U-Net, Transformer, etc.) without the encoder-decoder bottleneck constraint of VAEs.

Diffusion models can be viewed as a hierarchical VAE with $T$ latent layers, where each layer's prior is conditioned on the layer above.

Beyond raw quality, diffusion models offer remarkable flexibility:

1. Conditional generation: Guide the denoising process with text, class labels, or other signals via classifier-free guidance. This enables text-to-image (Stable Diffusion, DALL-E 3), text-to-video, and text-to-audio.

2. Inpainting and editing: Start the reverse process from a partially noised version of an existing image to edit specific regions while preserving the rest.

3. Super-resolution: Condition the denoising on a low-resolution input to generate high-resolution outputs.

4. Latent space control: Latent diffusion models (Rombach et al., 2022) run the diffusion process in a compressed latent space for efficiency, enabling Stable Diffusion to run on consumer GPUs.

5. Theoretical connections: Diffusion models connect to score matching, stochastic differential equations, and optimal transport, providing rich theoretical foundations.