Chapter 13: Generative & Production ML

Labels are expensive, but an input can serve as its own target. The encoder $f_\theta$ maps an input $x$ with $D$ features to a code $z$ with $d$ features; the decoder $g_\phi$ maps $z$ back to a reconstruction $\hat{x}$ with $D$ features. Training minimizes a reconstruction loss such as mean squared error, or binary cross-entropy when pixels lie in [0, 1]. With linear layers and squared error, the best autoencoder spans the same subspace as PCA's top $d$ components, so nonlinear layers are what let it follow curved data manifolds. The code becomes a feature vector for later tasks such as clustering or classification.

Reconstruction alone teaches nothing if copying is allowed. When the code dimension $d$ is at least the input dimension $D$, the network can learn the identity map: perfect reconstruction with features that are no more useful than raw pixels. A bottleneck with $d$ much smaller than $D$ forces the encoder to keep only the information that best explains the data and to discard noise and redundancy. Choosing $d$ is a trade-off: too small loses real structure and blurs reconstructions; too large drifts back toward copying. Reconstruction error always falls as $d$ grows, so pick $d$ by downstream usefulness, not by reconstruction loss alone.

A denoising autoencoder blocks copying without a narrow code. Each training input $x$ is corrupted to $\tilde{x}$, for example by zeroing a random 30 percent of pixels or adding Gaussian noise, and the network must output the clean $x$. Copying $\tilde{x}$ now gives a poor loss, so the network must learn how pixels depend on one another in order to fill in the gaps. Vincent and colleagues showed that stacked denoising features pre-trained deep networks well. The noise level is a hyperparameter: too little allows near-copying, while too much destroys the information needed. Later, denoising score matching connected this idea to diffusion models.

A sparse autoencoder can use a code wider than the input, called overcomplete, yet still avoid copying, because a penalty keeps most code units inactive for any given input. One form adds $\lambda$ times the sum over units $j$ of $\mathrm{KL}(\rho \,\Vert\, \hat{\rho}_j)$, where $\hat{\rho}_j$ is unit $j$'s average activation over the data and $\rho$ is a small target such as 0.05; another adds an L1 penalty on activations. Each unit then specializes in a pattern that appears in only a few inputs, which tends to make units more interpretable. The penalty weight $\lambda$ trades reconstruction for sparsity; set too high, units go permanently dead.

A plain encoder maps each input to one point, so the decoder never learns what nearby points mean. A VAE encoder instead outputs the parameters of a Gaussian, $q_\phi(z \mid x) = \mathcal{N}(\mu_\phi(x), \sigma^2_\phi(x))$: a mean vector $\mu$ and a standard deviation vector $\sigma$, one entry per latent dimension. In practice it outputs $\log \sigma^2$ for numerical stability. During training the decoder sees a different sample $z$ on each pass, so it must decode a whole neighborhood around $\mu$ to the same input. That smears each input over a region, filling the gaps between training points. When $\sigma$ collapses toward 0, the VAE degenerates into an ordinary autoencoder.

We want to maximize the data likelihood $\log p_\theta(x)$, but it requires integrating over every $z$, which is intractable. The evidence lower bound replaces it: $\mathbb{E}_{q}[\log p_\theta(x \mid z)] - D_{KL}(q_\phi(z \mid x) \,\Vert\, p(z))$. The first term rewards accurate reconstruction of $x$ from sampled codes. The second penalizes each encoded Gaussian for straying from the prior $\mathcal{N}(0, I)$; for Gaussians it has the closed form one half times the sum of $\mu^2 + \sigma^2 - \log \sigma^2 - 1$. The gap between ELBO and true likelihood is the KL to the true posterior. If KL dominates, the encoder ignores $x$, called posterior collapse; beta-VAE reweights the KL deliberately.

Training a VAE needs the gradient of the reconstruction loss with respect to $\mu$ and $\sigma$, but $z$ is drawn at random, and you cannot differentiate through the act of sampling. The reparameterization trick rewrites the sample as $z = \mu + \sigma \odot \epsilon$ with $\epsilon \sim \mathcal{N}(0, I)$ drawn independently of the parameters. Now $z$ is a deterministic, differentiable function of $\mu$ and $\sigma$, and the randomness enters like an extra input. Gradients flow as usual: $\partial z / \partial \mu = 1$ and $\partial z / \partial \sigma = \epsilon$. The alternative, the score-function or REINFORCE estimator, is unbiased but has far higher variance, which makes training slow.

Once trained, the encoder is no longer needed to create data. Sample $z \sim \mathcal{N}(0, I)$, pass it through the decoder, and you get a new example; this works because the KL term made the encoded distributions cover the prior. Interpolating between two codes, $z_t = (1 - t) z_1 + t z_2$ for $t$ from 0 to 1, yields a smooth sequence of plausible outputs, and adding attribute directions, such as the average code of smiling faces minus that of neutral faces, can edit images. VAE samples are often blurry, because a Gaussian or squared-error decoder averages over the plausible outputs for each code.

A GAN replaces a hand-written loss with a learned critic. The discriminator $D$ outputs the probability that its input is real; the generator $G$ maps noise $z$ to a sample. $D$ maximizes $\mathbb{E}[\log D(x)] + \mathbb{E}[\log(1 - D(G(z)))]$, which is ordinary binary cross-entropy for real versus fake, while $G$ minimizes the same quantity. For a fixed $G$, the best discriminator is $p_{\text{data}} / (p_{\text{data}} + p_G)$, and plugging it in shows that $G$ minimizes the Jensen-Shannon divergence between the two distributions. At the global optimum $p_G = p_{\text{data}}$, $D$ outputs 0.5 everywhere, and the objective equals $-\log 4$.

GAN training alternates two gradient steps on each minibatch. First update $D$ on a batch of real samples labeled 1 and generated samples labeled 0. Then update $G$, holding $D$ fixed, to make $D$ score its samples as real. The original paper allows $k$ discriminator steps per generator step and used $k = 1$. A trap appears early: when fakes are poor, $D(G(z))$ is near 0, and the gradient of $\log(1 - D(G(z)))$ with respect to $D$'s logit is just $-D(G(z))$, nearly zero. The fix is the non-saturating loss: $G$ maximizes $\log D(G(z))$, which has the same fixed point but strong gradients when $G$ is losing.

Mode collapse is the most common GAN failure: the generator maps many different noise vectors to the same few outputs, covering only part of the data distribution. It happens because $G$ is updated against the current $D$ only; if one kind of sample fools $D$ today, every $z$ moves toward it. $D$ then learns to reject that kind, $G$ jumps to another mode, and the pair can cycle without converging. Each sample may look sharp, so per-sample quality hides the problem. Fixes include minibatch discrimination, where $D$ sees statistics across a whole batch, feature matching, unrolled GANs, and Wasserstein losses.

Variants fix specific weaknesses of the original GAN. DCGAN set the convolutional recipe: strided convolutions instead of pooling, batch normalization, ReLU in the generator, and LeakyReLU in the discriminator. WGAN replaces Jensen-Shannon divergence with the Wasserstein, or earth mover's, distance, estimated by a critic that must be 1-Lipschitz, which was first enforced by weight clipping and later by a gradient penalty in WGAN-GP. Its loss tracks sample quality, so it doubles as a training curve. StyleGAN feeds a mapped latent code into every layer, giving control over coarse and fine features; BigGAN scales batch size and class conditioning. Diffusion models have since overtaken GANs for most image generation.

A training run is defined by far more than its code: hyperparameters, data version, random seed, library versions, and hardware all change the result. Experiment tracking logs each run as a record of parameters, metrics over time, artifacts such as weights and plots, the code commit, and a hash of the training data. Tools such as MLflow, Weights and Biases, and Neptune store these records in a queryable database with dashboards for comparing runs. Without tracking, a good result found in week two often cannot be reproduced in week ten. Log automatically from the training script; manual notes in notebooks are incomplete exactly when they matter.

A model registry is the version-controlled catalogue of trained models that are candidates for deployment. Each version records its lineage: the training run, code commit, data version, evaluation metrics, and who approved it. Versions move through stages such as staging, production, and archived, and serving systems load whatever version currently holds the production alias rather than a hard-coded file. That makes promotion and rollback a metadata change instead of a redeploy. Lineage also supports debugging and audits, since a bad prediction can be traced to the exact data and code behind it. Without a registry, teams lose track of which file is actually serving.

Serving turns a trained model into a dependable prediction service. Online serving answers single requests behind an API under a latency budget; batch serving scores large datasets on a schedule; edge serving runs on devices. Model servers such as TorchServe, Triton, TensorFlow Serving, and BentoML add dynamic batching, which waits a few milliseconds to group requests so the GPU works efficiently, as well as versioning and health checks. A load balancer spreads traffic across replicas and can split it between models for A/B tests or canary releases. Measure tail latency, p95 or p99, not the average, because the slowest requests are what users notice.

A deployed model degrades silently, because nothing crashes when the world changes. Data drift is a change in the input distribution $P(X)$, such as new merchant categories. Concept drift is a change in $P(Y \mid X)$: the same features now mean a different outcome, as when fraudsters change tactics. Data drift can be measured on unlabeled traffic with statistics such as the Population Stability Index or a Kolmogorov-Smirnov test per feature; concept drift needs labels, which often arrive weeks late. Monitor input statistics, the prediction distribution, and delayed accuracy, alert on thresholds, and retrain on fresh data when drift is persistent rather than a one-day spike.