CLIP

The labeling bottleneck:

Supervised learning requires manually labeled datasets. ImageNet took over 2.5 years and 49,000 workers on Amazon Mechanical Turk to label 1.4M images across 1,000 classes. Scaling to more classes or domains requires proportionally more labels — and the label taxonomy itself is a bottleneck. ImageNet's 1,000 classes are a specific, somewhat arbitrary slice of visual concepts (205 dog breeds but no "sunset" or "protest").

Why existing approaches fall short:

• Supervised classification: Fixed set of classes defined at training time. Adding a new class means collecting and labeling more data, redefining the softmax head, and retraining. Each dataset encodes a narrow taxonomy that limits what the model can recognize.
• Captioning models (VirTex): Predict exact captions word-by-word using autoregressive decoding. This is expensive per example (sequential token generation) and fragile — small vocabulary changes or paraphrasing break the training signal. VirTex trained on only 118K COCO images and showed limited transfer.
• ICMLM (Image-Conditioned Masked Language Model): Used a BERT-style masked language modeling objective conditioned on images. Trained on Conceptual Captions (3.3M pairs), it achieved only 11.5% zero-shot accuracy on ImageNet — far below usable performance.
• Bag-of-words approaches: Predict which words appear in the caption (multi-label classification over a fixed vocabulary). Better data efficiency than captioning but limited to a fixed vocabulary and can't capture word order or compositional meaning.

The data efficiency problem:

With N image-text pairs, a generative captioning model gets exactly N training examples — one gradient signal per pair. A bag-of-words model gets slightly more by predicting multiple words, but still N examples. For contrastive learning, every image is compared against every text in the batch, giving N² pairwise comparisons from just N examples. With a batch size of 32,768, that is over 1 billion comparisons per batch — an extraordinary increase in learning signal per data point.

The natural language advantage:

The internet contains billions of image-text pairs (alt text, captions, titles, descriptions). This supervision is free, diverse, and naturally describes visual concepts in open-ended language — not constrained to a fixed label set. A single caption like "a golden retriever playing fetch on the beach at sunset" simultaneously teaches the model about dogs, beaches, activities, lighting conditions, and their co-occurrence patterns. The OpenAI team estimated that over 400 million such pairs could be collected with sufficient quality filtering.

Contrastive pre-training learns by matching images to their captions within a batch, treating all non-matching pairs as negatives:

Step 1: Sample a batch of N = 32,768 image-text pairs $(x_1, t_1), ..., (x_N, t_N)$

Each pair consists of an image and its associated caption from the WIT dataset. The large batch size is critical: it provides 32,767 negative examples per positive pair, forcing the model to make fine-grained distinctions.

Step 2: Encode each image and text independently through their respective encoders and linear projections:

Image path: $224 \times 224 \times 3$ image $\rightarrow$ ViT-L/14 (24 blocks, 1024-d) $\rightarrow$ [CLS] token $\in \mathbb{R}^{1024}$ $\rightarrow$ linear projection $W_i \in \mathbb{R}^{512 \times 1024}$ $\rightarrow$ L2-normalize: $z_i^{\text{img}} = \frac{W_i \cdot \text{ViT}(x_i)_{[\text{CLS}]}}{\|W_i \cdot \text{ViT}(x_i)_{[\text{CLS}]}\|} \in \mathbb{R}^{512}$

Text path: tokenized caption (max 76 BPE tokens) $\rightarrow$ Transformer (12 blocks, 512-d) $\rightarrow$ [EOS] token $\in \mathbb{R}^{512}$ $\rightarrow$ linear projection $W_t \in \mathbb{R}^{512 \times 512}$ $\rightarrow$ L2-normalize: $z_i^{\text{txt}} = \frac{W_t \cdot \text{Transformer}(t_i)_{[\text{EOS}]}}{\|W_t \cdot \text{Transformer}(t_i)_{[\text{EOS}]}\|} \in \mathbb{R}^{512}$

This produces two matrices of embeddings: $Z^{\text{img}} \in \mathbb{R}^{N \times 512}$ and $Z^{\text{txt}} \in \mathbb{R}^{N \times 512}$.

Step 3: Compute the NxN cosine similarity matrix scaled by a learned temperature: $s_{ij} = \frac{z_i^{\text{img}} \cdot z_j^{\text{txt}}}{\|z_i^{\text{img}}\| \, \|z_j^{\text{txt}}\|} \cdot \exp(\tau)$

Since embeddings are already L2-normalized, the dot product equals cosine similarity. The temperature $\tau$ is a learned log-parameterized scalar (initialized at $\log(1/0.07) \approx 2.66$) that controls the sharpness of the softmax distribution. It is clamped to prevent $\exp(\tau)$ from exceeding 100, which would cause training instability. The resulting matrix $S \in \mathbb{R}^{N \times N}$ has $N^2 = 1,073,741,824$ entries for $N = 32{,}768$.

Step 4: Symmetric contrastive loss (InfoNCE):

The diagonal entries $s_{ii}$ are the positive (matched) pairs. All off-diagonal entries are negatives. The loss treats each row as an N-way classification problem (which text matches this image?) and each column as the reverse (which image matches this text?):

$\mathcal{L}_{\text{img}} = -\frac{1}{N}\sum_{i=1}^{N} \log \frac{\exp(s_{ii})}{\sum_{j=1}^{N} \exp(s_{ij})}$

$\mathcal{L}_{\text{txt}} = -\frac{1}{N}\sum_{i=1}^{N} \log \frac{\exp(s_{ii})}{\sum_{j=1}^{N} \exp(s_{ji})}$

$\mathcal{L} = \frac{1}{2}(\mathcal{L}_{\text{img}} + \mathcal{L}_{\text{txt}})$

Why contrastive beats generative and bag-of-words:

The authors directly compared three objectives on the same data and compute budget. Contrastive learning achieved 4x higher zero-shot ImageNet accuracy than the equivalent predictive (captioning) model, reaching usable accuracy (over 60%) within the training budget while the generative approach plateaued around 16%. The bag-of-words contrastive approach was nearly as good but lost compositional understanding ("a dog biting a man" vs "a man biting a dog").

Why batch size matters:

With a batch size of 32,768, each example has 32,767 negatives. This massive negative set forces the model to learn fine-grained visual-semantic distinctions — it cannot succeed by learning coarse categories alone. The model must distinguish "a golden retriever on a beach" from "a labrador on a beach," "a golden retriever in a park," and 32,765 other captions. Smaller batch sizes (e.g., 256) dramatically reduce performance because the contrastive signal becomes too easy.

Data flow through contrastive pre-training (one step):

Batch of 32,768 image-text pairs $\downarrow$ Image branch: $[32768, 224, 224, 3]$ $\xrightarrow{\text{patchify}}$ $[32768, 196, 768]$ $\xrightarrow{\text{ViT-L/14 (24 blocks)}}$ $[32768, 1024]$ (CLS) $\xrightarrow{W_i}$ $[32768, 512]$ $\xrightarrow{\text{L2-norm}}$ $Z^{\text{img}}$ Text branch: $[32768, 76]$ tokens $\xrightarrow{\text{Transformer (12 blocks)}}$ $[32768, 512]$ (EOS) $\xrightarrow{W_t}$ $[32768, 512]$ $\xrightarrow{\text{L2-norm}}$ $Z^{\text{txt}}$ $\downarrow$ Similarity matrix: $S = Z^{\text{img}} \cdot (Z^{\text{txt}})^T \cdot \exp(\tau) \in \mathbb{R}^{32768 \times 32768}$ $\downarrow$ InfoNCE loss on rows (image-to-text) + columns (text-to-image) $\downarrow$ Gradients flow back through both encoders and projections