Week 5-6

Chapter 6: Latent Diffusion & Stable Diffusion

Running diffusion in pixel space is computationally prohibitive for high-resolution images. Latent Diffusion Models (LDMs) solve this by first compressing images into a compact latent space using a pretrained autoencoder, then running the diffusion process entirely in that latent space. This chapter covers the autoencoder architecture, the latent U-Net, cross-attention conditioning, text encoders, the full Stable Diffusion pipeline, and ControlNet for spatial conditioning.

Chapter Overview

Diffusion models operating directly in pixel space face a fundamental scaling problem: a 512×512 RGB image has 786,432 dimensions. Running hundreds of denoising steps in this space demands enormous computation and memory, limiting practical resolution and batch sizes.

Latent Diffusion Models (LDMs), introduced by Rombach et al. (2022), elegantly decouple perceptual compression from generative modeling. A pretrained autoencoder compresses images into a spatial latent representation that is typically $4\times$ to $8\times$ smaller per spatial dimension. The diffusion model then operates entirely in this latent space, reducing computation by orders of magnitude while preserving perceptual quality.

The key insight is that most of an image's bits encode imperceptible high-frequency detail. By learning a perceptual compression that discards these details, the diffusion model can focus its capacity on semantically meaningful structure---object composition, lighting, texture patterns---rather than wasting it on pixel-level noise.

Stable Diffusion is the most prominent LDM implementation, combining a KL-regularized autoencoder, a U-Net with cross-attention layers, and a CLIP or OpenCLIP text encoder. ControlNet extends this architecture by adding spatial conditioning (edges, depth maps, pose skeletons) through trainable copies of the U-Net encoder blocks.

This chapter covers:

Autoencoders for Latent Space: VQ-VAE and KL-regularized autoencoders for perceptual compression
LDM Architecture: The U-Net operating in latent space with computational advantages
Cross-Attention Conditioning: Injecting text and image conditions into the denoising process
Text Encoders: CLIP and T5 for mapping text to embedding space
The Stable Diffusion Pipeline: End-to-end generation from text prompt to image
ControlNet: Adding precise spatial control to pretrained diffusion models

Chapter Roadmap

Click any topic to jump in

Autoencoders

VQ-VAE and KL encoders — compressing images to latent space for 64× faster diffusion.

Diffusion in latent space

Architecture and conditioning

LDM Architecture

U-Net in latent space — the key insight making high-res generation practical.

Cross-Attention

Injecting text embeddings via cross-attention — how text controls generation.

The full system

Encoding and pipeline

Text Encoders

CLIP and T5 — converting prompts into embeddings that guide generation.

Stable Diffusion

The full pipeline: text encoder → U-Net → VAE decoder.

Spatial control

ControlNet

Spatial control via edges, depth, poses — conditioning without retraining.

Sign up to unlock this chapter

This chapter is part of PixelBank Premium. Create a free account, then upgrade to read the full lesson — concepts, walkthroughs, and exercises.

Sampling & Acceleration

Conditional Generation