SigLIP Pairwise Sigmoid Loss

mediumPROContrastive Objectives: CLIP and SigLIPVision-Language ModelsVLM 1: Vision-Language Models

Problem Statement

Implement SigLIP's sigmoid contrastive loss over a batch of image and text embeddings.

Background

SigLIP replaces CLIP's softmax with an independent binary classification per pair. Every one of the n x n image-text pairs gets a label

$z_{ij} = \begin{cases} +1 & i = j \\ -1 & i \neq j \end{cases}$

and a logit built from the scaled cosine similarity plus a learned bias b (initialised strongly negative, around -10, because the batch is overwhelmingly negatives):

$\ell_{ij} = s \cdot \hat I_i \cdot \hat T_j + b$

The loss is the mean over the batch of the summed pairwise logistic losses:

$\mathcal{L} = \frac{1}{n}\sum_{i=1}^{n}\sum_{j=1}^{n} \log\!\left(1 + e^{-z_{ij}\ell_{ij}}\right)$

Note the normalisation: the double sum is divided by n, not by n^2.

The point of the reformulation is that no term needs any other pair's logit. CLIP's denominator sums the whole row, forcing an all-gather across every device in the batch; SigLIP's terms are independent, so it trains at small batch sizes and shards trivially. That independence is why -log sigmoid must be computed pairwise here.

log(1 + exp(-z*l)) overflows for large negative z*l. Use numpy.logaddexp(0.0, -z*l), which is the stable softplus.

Your Task

Implement:

def siglip_loss(image_emb, text_emb, logit_scale, bias):

Return the scalar loss rounded to 4 decimals.

Input Format

image_emb, text_emb - n x d nested lists, unnormalised, row i a positive pair
logit_scale - positive float
bias - float, typically negative

Output Format

A single float rounded to 4 decimals.

Sample

img = [[1.0, 0.0], [0.0, 1.0]]
txt = [[1.0, 0.0], [0.0, 1.0]]
print(siglip_loss(img, txt, 10.0, -5.0))

Output:

0.0134

Example

Input

img = [[1.0, 0.0], [0.0, 1.0]]
txt = [[1.0, 0.0], [0.0, 1.0]]
print(siglip_loss(img, txt, 10.0, -5.0))

Output

0.0134

Reasoning

The cosine matrix is the identity. Diagonal logits are 101 - 5 = 5, giving softplus(-5) = 0.0067 each. Off-diagonal logits are 100 - 5 = -5 with label -1, so z*l = +5 and softplus(-5) = 0.0067 each. Four terms of 0.006715 summed and divided by n = 2 gives 0.0134.

Constraints

1 <= n <= 64, 1 <= d <= 64; no row is the zero vector
L2-normalise the embedding rows before taking dot products
Labels are +1 on the diagonal and -1 everywhere else
The double sum is divided by n (NOT by n * n)
Use np.logaddexp (stable softplus) - naive log(1 + exp(-x)) overflows
Round the returned loss to 4 decimals

Learn this in

Vision EncodersVLM Study Plan

More from VLM 1: Vision-Language Models

Visual Token Pruning by Attention Masshard CHAIR-i and CHAIR-s Hallucination Ratesmedium Merge a LoRA Update into the Base Weightmedium QLoRA Memory Footprint Estimatemedium POPE Precision, Recall, and F1medium

Concept primerPREVIEW

1. Background Knowledge

SigLIP (Sigmoid Loss for Language Image Pre-Training) is a variant of the contrastive learning objective used in Vision-Language Models (VLMs) like CLIP. While CLIP uses a softmax cross-entropy loss that requires normalizing logits across the entire batch (creating a dependency on all other samples), SigLIP reformulates the problem as independent binary classification for every possible image-text pair. This independence allows for more efficient distributed training and better performance with smaller batch sizes.

In this formulation, we treat every pair $(i, j)$ as a binary classification task. If the image $i$ and text $j$ are a matching pair ( $i=j$ ), the label $z_{ij}$ is $+1$ . If they are non-matching ( $i \neq j$ ), the label is $-1$ . The model predicts a logit $\ell_{ij}$ based on the scaled cosine similarity between the embeddings plus a learned bias term. The bias is typically initialized to a large negative value (e.g., -10) because the vast majority of pairs in a batch are negatives, and the model needs to start by predicting low probabilities for these mismatches.

The loss function is the mean of the binary cross-entropy (logistic loss) over all $n \times n$ pairs, normalized by $n$ (not $n^2$ ). Mathematically, this is expressed as:

\mathcal{L} = \frac{1}{n}\sum_{i=1}^{n}\sum_{j=1}^{n} \log\!\left(1 + e^{-z_{ij}\ell_{ij}}\right)

This formulation avoids the numerical instability of softmax and the communication overhead of all-reduce operations required for the denominator in CLIP.

2. Algorithm Approach

The core algorithm involves three main phases: normalization, logit computation, and stable loss aggregation.

Normalization: Convert raw embeddings into unit vectors to compute cosine similarity. This is done by dividing each vector by its L2 norm.
Logit Matrix Construction: Compute the dot product between normalized image and text embeddings to get cosine similarities. Scale these by logit_scale and add the bias to form the logit matrix $\ell$ .
Label Matrix Construction: Create a matrix where diagonal elements are $+1$ (positive pairs) and off-diagonal elements are $-1$ (negative pairs).
Stable Loss Calculation: Compute the logistic loss for each element. To prevent numerical overflow when $-z_{ij}\ell_{ij}$ is large and negative, use a numerically stable function like numpy.logaddexp(0.0, -z * l) instead of directly computing $\log(1 + \exp(\dots))$ .
Aggregation: Sum all losses and divide by $n$ (the batch size) to get the final scalar loss.

3. Step-by-Step Strategy

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results