Chapter 5: Contrastive Learning & CLIP

Given a batch of $N$ image-text pairs, InfoNCE treats each pair as a $N$-way classification problem: the correct text for image $i$ is class $i$. The probability is computed via softmax over scaled cosine similarities. The loss is the average cross-entropy over all $N$ samples, applied symmetrically for both image-to-text and text-to-image directions.

The temperature $\tau$ in InfoNCE controls the sharpness of the softmax distribution. Low temperature ($\tau \rightarrow 0$): the distribution becomes peaked, focusing entirely on the highest-similarity pair. Gradients are concentrated on hard negatives, but training can be unstable. High temperature ($\tau \rightarrow \infty$): the distribution becomes uniform, treating all negatives equally. Training is stable but the model cannot learn fine-grained distinctions. CLIP uses $\tau$ as a learnable parameter initialized to $0.07$, allowing the model to find the optimal sharpness during training.

InfoNCE provides a lower bound on the mutual information $I(X;Y)$ between the two modalities: $I(X;Y) \geq \log(N) - \mathcal{L}_{\text{InfoNCE}}$. This means minimizing the contrastive loss maximizes a lower bound on how much information the image representation captures about the text (and vice versa). The bound tightens with larger batch sizes $N$, which is why CLIP benefits enormously from large batches -- it can capture more mutual information.

CLIP was trained on a proprietary dataset of 400 million (image, text) pairs collected from the internet. The data was curated using a query list of 500,000 concepts drawn from Wikipedia and WordNet. For each concept, up to 20,000 image-text pairs were collected, ensuring broad coverage. The text is not clean captions but natural alt-text, titles, and descriptions -- noisy but diverse. This 'natural language supervision' signal is weaker per-sample than manually annotated labels but astronomically more abundant.

CLIP uses two separate encoders: a Vision Transformer (ViT) or Modified ResNet for images and a Transformer for text. Each encoder independently processes its input and produces a fixed-size embedding vector. A learned linear projection maps each encoder's output to the shared embedding space where cosine similarities are computed. The encoders never see each other's inputs directly -- there is no cross-attention. This separation enables efficient inference: image and text embeddings can be precomputed and cached independently.

CLIP computes the contrastive loss across the entire batch simultaneously. With batch size $N = 32,768$, each forward pass creates a $32,768 \times 32,768$ similarity matrix (over 1 billion entries). The loss is computed as symmetric cross-entropy: each image must identify its correct text among 32,767 distractors, and vice versa. This requires distributing the computation across hundreds of GPUs, with all-gather operations to synchronize embeddings. The computational cost is $O(N^2 D)$ for the similarity matrix, where $D$ is the embedding dimension.

In zero-shot CLIP, classification works by: (1) encode the image, (2) encode text descriptions of each class (e.g., 'a photo of a dog', 'a photo of a cat'), (3) compute cosine similarity between the image embedding and each text embedding, (4) predict the class with highest similarity. The text prompts effectively define the classification head. The model has 'seen' concepts like dogs and cats during pretraining (via image-text pairs mentioning them) but has never been trained as a classifier with explicit labels.

The choice of text prompt dramatically affects zero-shot performance. Simple class names ('dog') underperform compared to contextualized prompts ('a photo of a dog'). This is because CLIP was trained on natural text, not isolated words. The context 'a photo of' bridges the distribution gap between class labels and web captions. Prompt ensembling averages predictions across multiple prompt templates ('a photo of a dog', 'a good photo of a dog', 'a drawing of a dog'), further improving robustness by up to 3-5% on benchmarks.

Instead of using a single text prompt per class, prompt ensembling generates multiple prompts from different templates and averages the resulting text embeddings. Given $K$ templates and $C$ classes, this creates $K \times C$ text embeddings. The averaged embedding per class is more robust to prompt wording choices. CLIP uses 80 templates including 'a photo of a {class}', 'a blurry photo of a {class}', 'a sculpture of a {class}', etc. This provides a consistent 2-5% accuracy improvement across benchmarks.

ALIGN (A Large-scale ImaGe and Text) used 1.8 billion image-text pairs with minimal filtering -- only a basic frequency-based filter to remove rare texts and near-duplicate images. Despite the noise (many pairs are weakly correlated), ALIGN matched CLIP's performance by relying on sheer scale. The key insight: for contrastive learning, data quantity can compensate for data quality because the loss function only needs relative rankings, not exact correspondences.

Florence extends contrastive learning with a hierarchical approach to visual representation. Instead of a single global embedding per image, Florence learns representations at multiple granularity levels: scene-level (coarse), object-level (medium), and region-level (fine). This enables the model to perform diverse downstream tasks from image classification (scene-level) to object detection (region-level) without task-specific architectures. Florence also uses a unified adapter framework for efficient fine-tuning across tasks.

EVA-CLIP achieves CLIP-level performance with significantly less compute by using better training recipes. Key innovations: (1) Masked image modeling (MIM) pretraining of the vision encoder before contrastive training, so it starts from a stronger initialization. (2) LAMB optimizer for more stable large-batch training. (3) Progressive training: start with low resolution and small model, gradually increase both. EVA-CLIP-18B is the largest open CLIP model with 18 billion parameters, achieving state-of-the-art zero-shot performance.

Contrastive models treat text more like a bag of words than an ordered sequence for alignment purposes. The caption 'a red car on a blue road' and 'a blue car on a red road' produce nearly identical text embeddings because they contain the same words. The global pooling that produces a single embedding discards word order and compositional structure. This means CLIP cannot reliably distinguish images where the same objects are present but in different configurations.

CLIP struggles with negation: 'a photo with no people' and 'a photo with people' produce similar embeddings. The contrastive objective does not explicitly learn logical operators like negation because web captions rarely use negation ('no people in this image' is uncommon vs 'people in this image'). Negation requires understanding what is absent, while contrastive learning is optimized for identifying what is present.

Attribute binding refers to correctly associating adjectives with their nouns: in 'a red car and a blue truck', 'red' binds to 'car' and 'blue' binds to 'truck'. CLIP frequently confuses these bindings because its global embedding merges all attributes and objects into one vector. The Winoground benchmark specifically tests this: it presents two images and two captions where the same entities appear with swapped attributes, and CLIP performs near chance level.