Loading...
Master the contrastive learning paradigm that revolutionized vision-language models. From the theoretical foundations of noise contrastive estimation through the full CLIP training pipeline, zero-shot transfer, and scaling variants like ALIGN and Florence, to the fundamental limitations that motivate generative approaches.
Contrastive learning is the engine that powers modern vision-language alignment. Rather than learning to generate text from images (expensive, slow), contrastive methods learn a shared embedding space where matched image-text pairs are close and unmatched pairs are far apart. This elegant formulation scales to billions of pairs scraped from the web.
CLIP (Contrastive Language-Image Pre-training), published by OpenAI in January 2021, demonstrated that a model trained on 400 million image-text pairs could match or exceed supervised models on dozens of classification benchmarks -- without seeing a single labeled example from those datasets. This "zero-shot" capability fundamentally changed how we think about visual recognition.
The key insight is deceptively simple: instead of training a classifier with a fixed set of classes, train an image encoder and a text encoder to agree on which images go with which captions. At inference time, you can classify images into any set of categories just by computing similarities with text descriptions of those categories.
This chapter builds the full picture:
Click any topic to jump in
How pulling positives together and pushing negatives apart creates structured embedding spaces — the core paradigm behind vision-language alignment.
The mathematical objective connecting contrastive learning to mutual information estimation, with temperature as the key control parameter.
Scaling contrastive learning to 400M web-scraped pairs with dual encoders and massive batch sizes — from theory to breakthrough.
What CLIP enables vs. how others scale the idea
Using text prompts as flexible classifiers — matching supervised models on dozens of benchmarks without a single labeled example.
How ALIGN, Florence, and EVA-CLIP push the recipe further with noisier data, hierarchical features, and efficient initialization.
Why contrastive models fail at compositionality, negation, and attribute binding — the failures that motivate generative approaches.
This chapter is part of PixelBank Premium. Create a free account, then upgrade to read the full lesson — concepts, walkthroughs, and exercises.