Chapter 5: Contrastive Learning & CLIP
Master the contrastive learning paradigm that revolutionized vision-language models. From the theoretical foundations of noise contrastive estimation through the full CLIP training pipeline, zero-shot transfer, and scaling variants like ALIGN and Florence, to the fundamental limitations that motivate generative approaches.
Chapter Overview
Contrastive learning is the engine that powers modern vision-language alignment. Rather than learning to generate text from images (expensive, slow), contrastive methods learn a shared embedding space where matched image-text pairs are close and unmatched pairs are far apart. This elegant formulation scales to billions of pairs scraped from the web.
CLIP (Contrastive Language-Image Pre-training), published by OpenAI in January 2021, demonstrated that a model trained on 400 million image-text pairs could match or exceed supervised models on dozens of classification benchmarks -- without seeing a single labeled example from those datasets. This "zero-shot" capability fundamentally changed how we think about visual recognition.
The key insight is deceptively simple: instead of training a classifier with a fixed set of classes, train an image encoder and a text encoder to agree on which images go with which captions. At inference time, you can classify images into any set of categories just by computing similarities with text descriptions of those categories.
This chapter builds the full picture:
- Contrastive learning foundations: Why pulling positives together and pushing negatives apart produces useful representations
- InfoNCE loss: The mathematical core of CLIP, connecting contrastive learning to mutual information
- CLIP training at scale: How 400M web-scraped pairs, dual encoders, and massive batch sizes come together
- Zero-shot transfer: Using language as a flexible classifier interface
- Scaling variants: ALIGN (1.8B noisy pairs), Florence (hierarchical features), EVA-CLIP
- Fundamental limitations: Why contrastive models fail at compositionality and what this means for VLMs
Chapter Roadmap
Click any topic to jump in
Contrastive Foundations
How pulling positives together and pushing negatives apart creates structured embedding spaces — the core paradigm behind vision-language alignment.
InfoNCE Loss
The mathematical objective connecting contrastive learning to mutual information estimation, with temperature as the key control parameter.
CLIP Training
Scaling contrastive learning to 400M web-scraped pairs with dual encoders and massive batch sizes — from theory to breakthrough.
What CLIP enables vs. how others scale the idea
Zero-Shot Transfer
Using text prompts as flexible classifiers — matching supervised models on dozens of benchmarks without a single labeled example.
Scaling Variants
How ALIGN, Florence, and EVA-CLIP push the recipe further with noisier data, hierarchical features, and efficient initialization.
Fundamental Limitations
Why contrastive models fail at compositionality, negation, and attribute binding — the failures that motivate generative approaches.
Sign up to unlock this chapter
This chapter is part of PixelBank Premium. Create a free account, then upgrade to read the full lesson — concepts, walkthroughs, and exercises.