Softmax vs Sigmoid: Batch Dependence of the Match Probability

Problem Statement

Show, numerically, the structural difference between CLIP and SigLIP: CLIP's match probability for a pair depends on which other examples share the batch, SigLIP's does not. Score the same pairs twice - once in the full batch, once after the batch is truncated - under both objectives.

Background

Both models score a pair with a scaled cosine similarity. They differ only in how that score becomes a probability.

CLIP normalises across the row - the denominator sums over every text in the batch:

$p^{\text{CLIP}}_{ii} = \frac{e^{s\,\hat I_i\cdot\hat T_i}}{\sum_{j=1}^{n} e^{s\,\hat I_i\cdot\hat T_j}}$

SigLIP normalises nothing - each pair is its own logistic regression, with a learned bias:

$p^{\text{SigLIP}}_{ii} = \sigma\!\left(s\,\hat I_i\cdot\hat T_i + b\right), \qquad \sigma(x) = \frac{1}{1 + e^{-x}}$

Now keep only the first k rows and columns and rescore those same k pairs. Every CLIP denominator has lost n - k terms, so every p^CLIP_ii moves - the identical image-caption pair is scored differently purely because its batch-mates went away. The SigLIP values are bit-for-bit unchanged: no term in p^SigLIP_ii mentions any other example.

That is the whole argument for the sigmoid loss. CLIP's gradient couples the batch, which is why it needs enormous batches and an all-gather of logits across devices; SigLIP's does not, so it trains as happily at batch 4k as at 32k and shards without cross-device communication.

Your Task

Implement:

def compare_objectives(image_emb, text_emb, logit_scale, bias, subset_size):

Every returned value is a mean over the first subset_size diagonal pairs - the same pairs in all four cases. What changes is the context:

• clip_full - CLIP probability computed with the whole n x n logit matrix, then averaged over the first subset_size diagonal entries
• clip_subset - CLIP probability computed after slicing both embedding matrices to their first subset_size rows, averaged over the diagonal
• siglip_full, siglip_subset - the same two contexts under the sigmoid probability

Return [clip_full, clip_subset, siglip_full, siglip_subset], each rounded to 4 decimals.

Input Format

• image_emb, text_emb - n x d nested lists, unnormalised
• logit_scale - positive float
• bias - float, used only by the SigLIP probability
• subset_size - integer with 1 <= subset_size <= n

Output Format

A list of four floats rounded to 4 decimals. The last two are always equal - that is the point of the exercise.

Sample

img = [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]
txt = [[1.0, 0.1], [0.1, 1.0], [1.0, 0.9]]
print(compare_objectives(img, txt, 5.0, -2.0, 2))

Output:

[0.8002, 0.9888, 0.9514, 0.9514]