SigLIP Pairwise Sigmoid Loss

Problem Statement

Implement SigLIP's sigmoid contrastive loss over a batch of image and text embeddings.

Background

SigLIP replaces CLIP's softmax with an independent binary classification per pair. Every one of the n x n image-text pairs gets a label

$z_{ij} = \begin{cases} +1 & i = j \\ -1 & i \neq j \end{cases}$

and a logit built from the scaled cosine similarity plus a learned bias b (initialised strongly negative, around -10, because the batch is overwhelmingly negatives):

$\ell_{ij} = s \cdot \hat I_i \cdot \hat T_j + b$

The loss is the mean over the batch of the summed pairwise logistic losses:

$\mathcal{L} = \frac{1}{n}\sum_{i=1}^{n}\sum_{j=1}^{n} \log\!\left(1 + e^{-z_{ij}\ell_{ij}}\right)$

Note the normalisation: the double sum is divided by n, not by n^2.

The point of the reformulation is that no term needs any other pair's logit. CLIP's denominator sums the whole row, forcing an all-gather across every device in the batch; SigLIP's terms are independent, so it trains at small batch sizes and shards trivially. That independence is why -log sigmoid must be computed pairwise here.

log(1 + exp(-z*l)) overflows for large negative z*l. Use numpy.logaddexp(0.0, -z*l), which is the stable softplus.

Your Task

Implement:

def siglip_loss(image_emb, text_emb, logit_scale, bias):

Return the scalar loss rounded to 4 decimals.

Input Format

• image_emb, text_emb - n x d nested lists, unnormalised, row i a positive pair
• logit_scale - positive float
• bias - float, typically negative

Output Format

A single float rounded to 4 decimals.

Sample

img = [[1.0, 0.0], [0.0, 1.0]]
txt = [[1.0, 0.0], [0.0, 1.0]]
print(siglip_loss(img, txt, 10.0, -5.0))

Output:

0.0134