Loading...
Running diffusion in pixel space is computationally prohibitive for high-resolution images. Latent Diffusion Models (LDMs) solve this by first compressing images into a compact latent space using a pretrained autoencoder, then running the diffusion process entirely in that latent space. This chapter covers the autoencoder architecture, the latent U-Net, cross-attention conditioning, text encoders, the full Stable Diffusion pipeline, and ControlNet for spatial conditioning.
Diffusion models operating directly in pixel space face a fundamental scaling problem: a 512×512 RGB image has 786,432 dimensions. Running hundreds of denoising steps in this space demands enormous computation and memory, limiting practical resolution and batch sizes.
Latent Diffusion Models (LDMs), introduced by Rombach et al. (2022), elegantly decouple perceptual compression from generative modeling. A pretrained autoencoder compresses images into a spatial latent representation that is typically to smaller per spatial dimension. The diffusion model then operates entirely in this latent space, reducing computation by orders of magnitude while preserving perceptual quality.
The key insight is that most of an image's bits encode imperceptible high-frequency detail. By learning a perceptual compression that discards these details, the diffusion model can focus its capacity on semantically meaningful structure---object composition, lighting, texture patterns---rather than wasting it on pixel-level noise.
Stable Diffusion is the most prominent LDM implementation, combining a KL-regularized autoencoder, a U-Net with cross-attention layers, and a CLIP or OpenCLIP text encoder. ControlNet extends this architecture by adding spatial conditioning (edges, depth maps, pose skeletons) through trainable copies of the U-Net encoder blocks.
This chapter covers:
Click any topic to jump in
VQ-VAE and KL encoders — compressing images to latent space for 64× faster diffusion.
Architecture and conditioning
U-Net in latent space — the key insight making high-res generation practical.
Injecting text embeddings via cross-attention — how text controls generation.
Encoding and pipeline
CLIP and T5 — converting prompts into embeddings that guide generation.
The full pipeline: text encoder → U-Net → VAE decoder.
Spatial control via edges, depth, poses — conditioning without retraining.
This chapter is part of PixelBank Premium. Create a free account, then upgrade to read the full lesson — concepts, walkthroughs, and exercises.