📘

Softmax vs Sigmoid: Batch Dependence of the Match Probability

HardVision-Language Models 📚 VLM 1: Vision-Language Models

Problem Statement

Show, numerically, the structural difference between CLIP and SigLIP: CLIP's match probability for a pair depends on which other examples share the batch, SigLIP's does not. Score the same pairs twice - once in the full batch, once after the batch is truncated - under both objectives.

Background

Both models score a pair with a scaled cosine similarity. They differ only in how that score becomes a probability.

CLIP normalises across the row - the denominator sums over every text in the batch:

$p^{\text{CLIP}}_{ii} = \frac{e^{s\,\hat I_i\cdot\hat T_i}}{\sum_{j=1}^{n} e^{s\,\hat I_i\cdot\hat T_j}}$

SigLIP normalises nothing - each pair is its own logistic regression, with a learned bias:

$p^{\text{SigLIP}}_{ii} = \sigma\!\left(s\,\hat I_i\cdot\hat T_i + b\right), \qquad \sigma(x) = \frac{1}{1 + e^{-x}}$

Now keep only the first k rows and columns and rescore those same k pairs. Every CLIP denominator has lost n - k terms, so every p^CLIP_ii moves - the identical image-caption pair is scored differently purely because its batch-mates went away. The SigLIP values are bit-for-bit unchanged: no term in p^SigLIP_ii mentions any other example.

That is the whole argument for the sigmoid loss. CLIP's gradient couples the batch, which is why it needs enormous batches and an all-gather of logits across devices; SigLIP's does not, so it trains as happily at batch 4k as at 32k and shards without cross-device communication.

Your Task

Implement:

def compare_objectives(image_emb, text_emb, logit_scale, bias, subset_size):

Every returned value is a mean over the first subset_size diagonal pairs - the same pairs in all four cases. What changes is the context:

clip_full - CLIP probability computed with the whole n x n logit matrix, then averaged over the first subset_size diagonal entries
clip_subset - CLIP probability computed after slicing both embedding matrices to their first subset_size rows, averaged over the diagonal
siglip_full, siglip_subset - the same two contexts under the sigmoid probability

Return [clip_full, clip_subset, siglip_full, siglip_subset], each rounded to 4 decimals.

Input Format

image_emb, text_emb - n x d nested lists, unnormalised
logit_scale - positive float
bias - float, used only by the SigLIP probability
subset_size - integer with 1 <= subset_size <= n

Output Format

A list of four floats rounded to 4 decimals. The last two are always equal - that is the point of the exercise.

Sample

img = [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]
txt = [[1.0, 0.1], [0.1, 1.0], [1.0, 0.9]]
print(compare_objectives(img, txt, 5.0, -2.0, 2))

Output:

[0.8002, 0.9888, 0.9514, 0.9514]

Example:

Input:

img = [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]
txt = [[1.0, 0.1], [0.1, 1.0], [1.0, 0.9]]
print(compare_objectives(img, txt, 5.0, -2.0, 2))

Output:

[0.8002, 0.9888, 0.9514, 0.9514]

Reasoning:

Pairs 0 and 1 are scored in both settings. Dropping the third pair removes one competitor from each CLIP row, so their mean softmax probability jumps from 0.8002 to 0.9888 even though no embedding changed. The SigLIP probability of a pair reads only that pair's own similarity, so it is 0.9514 in both settings.

Constraints:

1 <= n <= 64, 1 <= d <= 64; no row is the zero vector
L2-normalise embeddings before the dot product
The CLIP probability is a ROW-wise softmax (image-to-text direction only), computed stably
The SigLIP probability applies the bias; the CLIP probability does NOT
ALL FOUR values average over the SAME first subset_size diagonal pairs
Truncating the batch slices both embedding matrices to their first subset_size rows
Round each value to 4 decimals

More from VLM 1: Vision-Language Models

Visual Token Pruning by Attention MassHard CHAIR-i and CHAIR-s Hallucination RatesMedium Merge a LoRA Update into the Base WeightMedium QLoRA Memory Footprint EstimateMedium POPE Precision, Recall, and F1Medium

1. Background Knowledge

This problem explores the fundamental difference in batch dependence between two popular contrastive learning objectives: CLIP (Contrastive Language-Image Pre-training) and SigLIP (Sigmoid Loss for Language Image Pre-Training). Understanding this distinction is crucial for designing scalable vision-language models.

CLIP uses a softmax loss over the entire batch. The probability of a correct image-text pair is calculated by normalizing the exponential of its similarity score against the sum of exponentials of all other text embeddings in the batch. This creates a coupling effect: the probability of a specific pair depends on the "competition" from other pairs in the same batch. If you remove other examples, the denominator shrinks, changing the probability even if the target pair's score remains identical. This necessitates large batches and complex distributed training (all-gather operations) to maintain consistent gradients.

SigLIP, conversely, uses a sigmoid loss. It treats each pair as an independent binary classification task (positive vs. negative), often with a learned bias term. The probability is computed using the logistic function $\sigma(x) = \frac{1}{1 + e^{-x}}$ . Crucially, this calculation involves only the specific image-text pair and the global parameters (scale and bias). It does not reference other examples in the batch. Therefore, the probability for a pair is invariant to the presence or absence of other batch members. This allows SigLIP to train efficiently with smaller batches and without cross-device communication for logits.

2. Algorithm Approach

The core task is to implement two distinct probability calculation functions and apply them under two different data contexts (full batch vs. subset).

Normalization: Both methods require L2-normalized embeddings to compute cosine similarity. You must normalize the input image_emb and text_emb matrices first.
Similarity Matrix: Compute the dot product between normalized image and text embeddings to get the raw similarity scores.
CLIP Probability: Implement the softmax normalization. For each row $i$ , the probability is $e^{s \cdot \text{sim}_{ij}} / \sum_{k} e^{s \cdot \text{sim}_{ik}}$ .
SigLIP Probability: Implement the sigmoid function with scale and bias: $\sigma(s \cdot \text{sim}_{ii} + b)$ .
Context Switching: Calculate these probabilities twice for each method: once using the full $n \times n$ matrix, and once using only the first subset_size rows/columns.
Aggregation: Average the diagonal probabilities for the first subset_size pairs in each case.

3. Step-by-Step Strategy

Normalize Embeddings:

Iterate through image_emb and text_emb.
For each vector, compute its L2 norm ( $\sqrt{\sum x_i^2}$ ).
Divide each element by its norm. Store these as img_norm and txt_norm.

Define Helper Functions:

compute_clip_probs(img_mat, txt_mat, scale):
Compute the dot product matrix $S = \text{img\_mat} \times \text{txt\_mat}^T$ .
Multiply by scale.
For each row $i$ , compute the softmax: $P_{ii} = \exp(S_{ii}) / \sum_j \exp(S_{ij})$ .
Return the list of diagonal probabilities.
compute_siglip_probs(img_mat, txt_mat, scale, bias):
Compute the dot product matrix $S$ .
For each diagonal element $S_{ii}$ , compute $\sigma(\text{scale} \cdot S_{ii} + \text{bias})$ .
Return the list of diagonal probabilities.

Compute Full Batch Metrics:

Use the full img_norm and txt_norm.
Call compute_clip_probs to get all diagonal probabilities. Take the first subset_size and average them -> clip_full.
Call compute_siglip_probs to get all diagonal probabilities. Take the first subset_size and average them -> siglip_full.

Compute Subset Metrics:

Slice img_norm and txt_norm to keep only the first subset_size rows.
Call compute_clip_probs on these smaller matrices. Average the resulting diagonal probabilities -> clip_subset.
Call compute_siglip_probs on these smaller matrices. Average the resulting diagonal probabilities -> siglip_subset.

Final Output:

Round all four averages to 4 decimal places.
Return [clip_full, clip_subset, siglip_full, siglip_subset].

4. Common Pitfalls

Forgetting Normalization: The problem states inputs are "unnormalised". Cosine similarity requires unit vectors. Failing to normalize will yield incorrect dot products.
Softmax Numerical Stability: When computing $e^{s \cdot \text{sim}}$ , large values can cause overflow. While less critical for small batches in this specific problem, it's good practice to subtract the max value in each row before exponentiating: $\text{softmax}(x)_i = \frac{e^{x_i - \max(x)}}{\sum e^{x_j - \max(x)}}$ .
Indexing Errors: Ensure you are averaging only the diagonal elements (where image index equals text index) for the first subset_size pairs.
SigLIP Bias Application: Remember that the bias b is added after scaling the similarity but before the sigmoid function: $\sigma(s \cdot \text{sim} + b)$ .
Subset Slicing: When computing clip_subset, you must slice both image and text embeddings to subset_size. The resulting matrix is subset_size x subset_size. The denominator in CLIP's softmax will now sum over subset_size terms instead of n terms.

5. Time & Space Complexity

Time Complexity:
Normalization: $O(n \cdot d)$ , where $n$ is the batch size and $d$ is the embedding dimension.
Dot Product Matrix: $O(n^2 \cdot d)$ for the full batch.
Softmax/Sigmoid: $O(n^2)$ for the full batch.
Overall: Dominated by the matrix multiplication, $O(n^2 \cdot d)$ . Since we compute this twice (full and subset), the constant factor is small.
Space Complexity:
Storing normalized embeddings: $O(n \cdot d)$ .
Storing the similarity matrix: $O(n^2)$ .
Overall: $O(n^2 + n \cdot d)$ . For typical VLM dimensions where $d$ is large but $n$ is moderate, $O(n \cdot d)$ might dominate, but the $n^2$ term is significant for large batches.

solution.py

Python 3.13

Test Results

0/0

Run code to see test results.