Loading...
Understand the landscape of generative modeling: how machines learn to create new data, the fundamental difference between generative and discriminative approaches, and why diffusion models have emerged as the state-of-the-art framework for high-fidelity generation.
Generative models are at the heart of modern AI's creative capabilities. From photorealistic image synthesis to drug molecule design, these models learn the underlying distribution of data and generate novel samples that are statistically indistinguishable from real observations.
This chapter builds the conceptual foundation you need before diving into diffusion models specifically. We start with the fundamental distinction between generative and discriminative modeling, then survey the major families of generative models that have been developed over the past decade. You will learn how likelihood-based training works, why latent variables are essential for modeling complex data, and how researchers evaluate the quality and diversity of generated samples.
By the end of this chapter, you will understand:
Click any topic to jump in
The fundamental split — modeling data distributions vs learning decision boundaries.
GANs, VAEs, flows, autoregressive, diffusion — the landscape of generative approaches.
Direct likelihood vs latent variables
Normalizing flows and autoregressive models — directly maximizing log-likelihood.
VAEs and the ELBO — learning compressed representations through variational inference.
Evaluation and why diffusion wins
FID, IS, precision/recall — measuring quality and diversity of generated samples.
Training stability, mode coverage, sample quality — why diffusion surpassed GANs and VAEs.
The most fundamental divide in machine learning is between models that learn to classify existing data and models that learn to generate new data. Discriminative models learn the decision boundary --- given an input, what is the label? Generative models learn the full data distribution --- what does the data itself look like?
This distinction has profound implications for what a model can do, how it is trained, and how much data it needs. Generative modeling is strictly harder: if you know , you can derive via Bayes' rule, but the reverse is not true.
A discriminative model directly estimates the conditional probability of a label given input . It learns the decision boundary that separates classes without modeling how the data itself is generated. Examples include logistic regression, SVMs, and classification CNNs. These models are typically easier to train and achieve higher accuracy on supervised tasks because they focus only on the boundary, ignoring the internal structure of each class.
Discriminative models project high-dimensional input onto a decision surface. The conditional discards all information about the data manifold itself — it only needs the boundary where class posteriors cross . Statistically, this means fewer parameters to estimate and faster convergence rates, since the model ignores within-class variance entirely.
A CNN classifies images as cat or dog. Does it need to understand what cats look like in general, or just what distinguishes cats from dogs?
A generative model learns the full joint probability distribution of the data. Once learned, it can generate new samples by drawing from this distribution. The formula above shows the latent variable formulation: the data distribution is expressed as a marginalization over latent variables . Learning is fundamentally harder because the model must capture all variation in the data --- every texture, every pose, every lighting condition --- not just the features that distinguish classes.
Learning requires estimating density in the full ambient space, which is exponentially harder due to the curse of dimensionality. The integral marginalizes over latent factors, and tractability depends on whether this integral has closed form. When it does not, variational bounds or MCMC become necessary — the starting point for diffusion models.
If you have a generative model that has learned the distribution of face images, what tasks can it perform beyond generation?
The joint distribution contains complete information about both the data and labels. Generative models that learn can derive both (by marginalization) and (by conditioning). Conditional distributions capture only part of the picture. The chain rule of probability connects them: knowing any two of lets you derive the others. This is why generative models are more powerful but harder to train --- they must learn more information.
The chain rule reveals an information hierarchy: the joint contains strictly more information than any conditional. Bayes' theorem converts between conditionals, but requires the marginal — which is exactly the intractable quantity in generative modeling. This asymmetry is why generative models are universal but harder to train.
Given a dataset of handwritten digits with labels, compare what a generative vs discriminative model learns.
Use discriminative models when you have labeled data and only need predictions: classification, regression, ranking. They are more data-efficient for supervised tasks because they focus capacity on the decision boundary.
Use generative models when you need to: create new data (images, text, molecules), understand data structure, detect outliers, handle missing data, or perform semi-supervised learning. Generative models also enable data augmentation, privacy-preserving synthetic data, and creative applications.
In practice, the lines are blurring. Modern systems like CLIP combine discriminative and generative objectives, and diffusion models can be conditioned on text to bridge generation and classification.
From an information-theoretic view, discriminative models compress data into bits (the label), while generative models must represent the full entropy of the data distribution. When , the discriminative model is vastly more efficient for classification — but it cannot reconstruct, generate, or detect anomalies.
A hospital wants to (a) diagnose diseases from X-rays, and (b) generate synthetic X-rays for training. Which approach for each?
Prove that if you have a perfect generative model , you can always derive a perfect discriminative model , but the reverse is not true. What does this imply about the relative difficulty of generative vs discriminative learning?