Chapter 13: Generative & Production ML

An autoencoder minimizes reconstruction loss $L = \|\mathbf{x} - \hat{\mathbf{x}}\|^2$ where $\hat{\mathbf{x}} = g_\phi(f_\theta(\mathbf{x}))$, with encoder $f_\theta: \mathbb{R}^D \to \mathbb{R}^d$ and decoder $g_\phi: \mathbb{R}^d \to \mathbb{R}^D$ ($d \ll D$). For linear activations, the optimal encoder learns the PCA subspace — the columns of the encoder weight matrix span the same space as the top $d$ eigenvectors of the data covariance matrix. Non-linear activations enable capturing non-linear manifolds that PCA cannot represent.

The bottleneck dimension $d$ controls the information bottleneck: the encoder must discard $D - d$ dimensions of information. By the rate-distortion theorem, there exists a minimum distortion achievable at any given rate (bottleneck size). Setting $d$ too small loses important structure (underfitting); setting $d$ too large allows the identity mapping (no compression). Cross-validation on reconstruction error helps select $d$, but the optimal value depends on the intrinsic dimensionality of the data manifold.

A denoising autoencoder minimizes $L = \mathbb{E}_{\tilde{\mathbf{x}} \sim q(\tilde{\mathbf{x}}|\mathbf{x})} [\|\mathbf{x} - g(f(\tilde{\mathbf{x}}))\|^2]$ where $q$ is a corruption process (Gaussian noise, masking, salt-and-pepper). Vincent et al. showed this is equivalent to learning the score function $\nabla_{\mathbf{x}} \log p(\mathbf{x})$ — the gradient of the log data density — which connects denoising autoencoders to score-based diffusion models. The corruption prevents identity learning even when $d \geq D$.

Sparse autoencoders add a sparsity penalty $\Omega(\mathbf{h}) = \lambda \sum_j \text{KL}(\rho \| \hat{\rho}_j)$ where $\hat{\rho}_j = \frac{1}{N}\sum_i h_j(\mathbf{x}_i)$ is the average activation of neuron $j$ and $\rho \approx 0.05$ is the target sparsity. The KL divergence penalty drives most neurons to be inactive for most inputs, encouraging each neuron to specialize in a specific feature. This produces overcomplete representations ($d > D$) that are still useful because only a sparse subset activates for any given input.

The VAE maximizes the evidence lower bound: $\text{ELBO} = \mathbb{E}_{q_\phi(\mathbf{z}|\mathbf{x})}[\log p_\theta(\mathbf{x}|\mathbf{z})] - \text{KL}(q_\phi(\mathbf{z}|\mathbf{x}) \| p(\mathbf{z}))$. The first term is reconstruction quality; the KL term regularizes the approximate posterior $q_\phi$ to be close to the prior $\mathcal{N}(0, I)$. The reparameterization trick $\mathbf{z} = \mu + \sigma \odot \epsilon$ with $\epsilon \sim \mathcal{N}(0, I)$ enables backpropagation through the sampling step since the randomness is externalized.

For Gaussian $q_\phi(\mathbf{z}|\mathbf{x}) = \mathcal{N}(\mu, \text{diag}(\sigma^2))$ and prior $p(\mathbf{z}) = \mathcal{N}(0, I)$, the KL divergence has a closed form: $\text{KL} = \frac{1}{2} \sum_{j=1}^{d} (\mu_j^2 + \sigma_j^2 - \log \sigma_j^2 - 1)$. This penalizes latent dimensions that deviate from the standard normal — pushing means toward 0 and variances toward 1. The 'posterior collapse' problem occurs when the KL dominates and the encoder ignores the input entirely, outputting $q \approx p(\mathbf{z})$ for all inputs.

The VAE latent space is continuous and smooth: nearby points in $\mathbb{R}^d$ decode to similar outputs because the KL regularization prevents holes in the latent space. Linear interpolation $\mathbf{z}_t = (1-t)\mathbf{z}_1 + t\mathbf{z}_2$ between two encoded points produces a smooth transition in output space. The prior $p(\mathbf{z}) = \mathcal{N}(0, I)$ means we can generate new samples by simply drawing $\mathbf{z} \sim \mathcal{N}(0, I)$ and decoding — the KL term ensures the decoder sees this distribution during training.

A conditional VAE (CVAE) conditions both encoder and decoder on auxiliary information $\mathbf{c}$: $q_\phi(\mathbf{z}|\mathbf{x}, \mathbf{c})$ and $p_\theta(\mathbf{x}|\mathbf{z}, \mathbf{c})$. The ELBO becomes $\mathbb{E}_{q}[\log p(\mathbf{x}|\mathbf{z}, \mathbf{c})] - \text{KL}(q(\mathbf{z}|\mathbf{x}, \mathbf{c}) \| p(\mathbf{z}|\mathbf{c}))$. This enables controlled generation: for image generation conditioned on class labels, the CVAE learns to separate style ($\mathbf{z}$) from content ($\mathbf{c}$), allowing generation of new images for any specified class.

The GAN objective $\min_G \max_D \mathbb{E}_{\mathbf{x} \sim p_{\text{data}}} [\log D(\mathbf{x})] + \mathbb{E}_{\mathbf{z} \sim p_z} [\log(1 - D(G(\mathbf{z})))]$ has the global optimum $D^*(\mathbf{x}) = p_{\text{data}}(\mathbf{x}) / (p_{\text{data}}(\mathbf{x}) + p_G(\mathbf{x}))$. At equilibrium, $p_G = p_{\text{data}}$ and $D(\mathbf{x}) = 1/2$ everywhere. The minimax value at optimality equals $-\log 4$, and the divergence being minimized is the Jensen-Shannon divergence $\text{JSD}(p_{\text{data}} \| p_G)$.

GAN training alternates between $k$ steps of discriminator optimization and 1 step of generator optimization. The discriminator gradient is $\nabla_\phi [\log D_\phi(\mathbf{x}) + \log(1 - D_\phi(G_\theta(\mathbf{z})))]$, maximizing classification accuracy. The generator gradient uses the modified objective $-\nabla_\theta \log D_\phi(G_\theta(\mathbf{z}))$ instead of $\nabla_\theta \log(1 - D_\phi(G_\theta(\mathbf{z})))$ because the latter has vanishing gradients when $D$ is confident — the modified version provides stronger learning signal early in training.

Mode collapse occurs when $G$ maps many different $\mathbf{z}$ values to the same output region. Formally, the support of $p_G$ is much smaller than the support of $p_{\text{data}}$. This is a Nash equilibrium failure: G finds a mode that maximally fools D, and D cannot distinguish it from real data in that mode. Minibatch discrimination adds a feature $\|f(\mathbf{x}_i) - f(\mathbf{x}_j)\|$ that measures diversity within a batch, penalizing generators that produce identical outputs.

WGAN replaces the JS divergence with the Wasserstein-1 distance $W(p, q) = \inf_{\gamma \in \Pi(p,q)} \mathbb{E}_{(x,y) \sim \gamma} [\|x - y\|]$, estimated via the Kantorovich-Rubinstein dual: $W = \sup_{\|f\|_L \leq 1} \mathbb{E}_{p}[f(x)] - \mathbb{E}_{q}[f(x)]$. The Lipschitz constraint $\|f\|_L \leq 1$ is enforced via weight clipping or gradient penalty $\lambda \mathbb{E}[(\|\nabla f\| - 1)^2]$. Wasserstein distance provides gradients everywhere, even when distributions have non-overlapping support — the key advantage over JS divergence.

Experiment tracking logs the mapping $\text{run} \to (\theta, D, M, A)$ where $\theta$ is hyperparameters, $D$ is the data version hash, $M$ is metrics, and $A$ is artifacts (model weights, configs). The search space for hyperparameters grows as $O(k^d)$ for $d$ hyperparameters with $k$ values each (grid search) or is explored in $O(d / \epsilon^2)$ random trials to find an $\epsilon$-optimal configuration (random search). Bayesian optimization uses a Gaussian process surrogate to select the next trial in $O(n^3)$ for $n$ past observations, but amortizes this with better sample efficiency.

A model registry maintains a DAG (directed acyclic graph) of model lineage: $\text{model}_v \gets (\text{code}_{\text{commit}}, \text{data}_{\text{version}}, \text{hyperparams}, \text{metrics})$. Versioning enables reproducibility: any model can be re-created from its lineage. The staging workflow (development -> staging -> production -> archived) ensures only validated models serve traffic. Canary deployments route $p\%$ of traffic to the new model and monitor metrics before full rollout.

Serving latency decomposes as $t_{\text{total}} = t_{\text{network}} + t_{\text{preprocess}} + t_{\text{inference}} + t_{\text{postprocess}}$. For a transformer with $L$ layers, $d$ dimensions, and input length $n$: $t_{\text{inference}} = O(L \cdot n^2 \cdot d)$. Dynamic batching groups requests arriving within a time window $\Delta t$ to amortize GPU kernel launch overhead — optimal batch size balances throughput ($\propto B$) against latency ($\propto B$ for the first request in the batch). Quantization (INT8) reduces model size by $4\times$ and inference latency by $2$-$3\times$ with $< 1\%$ accuracy loss.

Data drift is detected by comparing feature distributions: Population Stability Index $\text{PSI} = \sum_i (p_i - q_i) \ln(p_i / q_i)$ measures divergence between training ($p$) and serving ($q$) distributions, with $\text{PSI} > 0.2$ indicating significant drift. Concept drift occurs when $P(Y|X)$ changes even if $P(X)$ is stable — detectable only with labeled data. Monitoring tracks prediction distribution entropy: a sudden increase suggests the model is uncertain about new data patterns. Automated retraining triggers when drift exceeds thresholds for $k$ consecutive measurement windows.