CLIP

CLIP (Contrastive Language-Image Pre-training) learns visual representations from natural language supervision, enabling zero-shot transfer to downstream tasks without any task-specific training data. Published by OpenAI in January 2021, it fundamentally changed how the field thinks about connecting vision and language.

The core idea replaces hand-labeled datasets (like ImageNet's 1.4M images across 1,000 classes, which took over 2.5 years of human annotation) with 400 million image-text pairs collected from the internet into a dataset called WebImageText (WIT). A dual-encoder architecture learns to align images and their captions in a shared 512-dimensional embedding space via a symmetric contrastive objective called InfoNCE. The image encoder (ViT-L/14 with 428M parameters) and the text encoder (a 63M-parameter Transformer) are trained jointly but process their inputs independently, producing embeddings that are compared via cosine similarity.

Key advances:

• Contrastive pre-training: Match images to captions in an NxN similarity matrix — N² training signal from just N examples. With batch size 32,768, each example is compared against 32,767 negatives per step
• Dual encoder architecture: Separate image encoder (ViT-L/14: 24 layers, 1024-d, 16 heads) and text encoder (12 layers, 512-d, 8 heads) mapping to a shared 512-d space via learned linear projections
• Zero-shot transfer: Classify any image using text prompts like "a photo of a {class}" — no labeled training data needed. Matches a fully supervised ResNet-50 on ImageNet at 76.2% top-1 accuracy
• Prompt engineering: 80 text templates per class, ensembled by averaging embeddings, improve accuracy by +3.5% (68.7% to 72.2%) through disambiguation and distribution coverage
• Scaling: Performance improves log-linearly with both compute and data. ViT-L/14 achieves 75.5% zero-shot on ImageNet; with 336px resolution, 76.2%. No saturation observed at 400M pairs
• Robust representations: Strong transfer across 27 diverse datasets spanning fine-grained recognition, scene classification, action recognition, satellite imagery, OCR, and texture classification

CLIP's zero-shot classifier matches a fully supervised ResNet-50 on ImageNet (76.2% top-1 accuracy) without seeing a single ImageNet training image. On 16 of 27 evaluation datasets, zero-shot CLIP outperforms supervised linear probes. The learned embeddings became the foundation for DALL-E 2, Stable Diffusion, and virtually every subsequent vision-language model. Training required 256 V100 GPUs for approximately 12 days, processing 12.8 billion image-text comparisons across 32 epochs.