SigLIP Loss with the Learnable Bias

mediumPROContrastive Objectives: CLIP and SigLIPVision-Language ModelsVLM 1: Vision-Language Models

Problem Statement

SigLIP replaces CLIP's softmax with an independent binary sigmoid loss per image-text pair, adding a learnable temperature t and bias b that correct for the extreme positive/negative imbalance. Implement it.

Background

For a batch of N pairs with cosine similarities sim (N x N), define the label matrix z_{ij} = +1 on the diagonal (matches) and z_{ij} = -1 off it (all other pairs are negatives). The SigLIP loss is

$\mathcal{L} = -\frac{1}{N}\sum_{i}\sum_{j} \log \sigma\!\big(z_{ij}\,(t\cdot \text{sim}_{ij} + b)\big)$

Note the normalization is by N (the batch size), not N^2. The bias b is initialized very negative (about -10) so training starts near the correct all-negative prior. Use log_sigmoid(x) = -softplus(-x) for stability.

Your Task

Implement:

def siglip_loss(sim, t, b):

Return the scalar loss rounded to 4 decimals.

Input Format

sim: N x N nested list of cosine similarities.
t (float): temperature/scale.
b (float): bias.

Output Format

A float rounded to 4 decimals.

Sample

print(siglip_loss([[1.0, 0.0], [0.0, 1.0]], 1.0, 0.0))

Output:

1.0064

Example

Input

print(siglip_loss([[1.0, 0.0], [0.0, 1.0]], 1.0, 0.0))

Output

1.0064

Reasoning

Logits t*sim+b are [[1,0],[0,1]]; with signs [[+,-],[-,+]] the diagonal terms use -log-sigmoid(1)=0.3133 and the off-diagonals -log-sigmoid(0)=0.6931. The four values sum to 2.0128; divided by N=2 gives 1.0064.

Constraints

1 <= N <= 512.
Labels are +1 on the diagonal, -1 elsewhere.
Sum over all N*N pairs but divide by N.
Use a numerically stable log_sigmoid; round to 4 decimals.

Learn this in

Vision EncodersVLM Study Plan Training Recipes: Pretraining, SFT & RLHFVLM Study Plan

More from VLM 1: Vision-Language Models

Visual Token Pruning by Attention Masshard CHAIR-i and CHAIR-s Hallucination Ratesmedium Merge a LoRA Update into the Base Weightmedium QLoRA Memory Footprint Estimatemedium POPE Precision, Recall, and F1medium

Concept primerPREVIEW

1. Background Knowledge

Contrastive learning in vision-language models (VLMs) aligns paired image and text embeddings by maximizing similarity for matching pairs while minimizing it for non-matching pairs. The original CLIP model uses a symmetric cross-entropy loss over a softmax distribution across the entire batch, treating each row (or column) of the similarity matrix as a multi-class classification problem where the correct match is the positive class.

SigLIP (Sigmoid Loss for Language Image Pre-Training) replaces this softmax with an independent binary sigmoid for every single pair in the $N \times N$ matrix. This decouples the computation: the probability of pair $(i, j)$ being a match is modeled independently of other pairs. The label matrix $Z$ is defined such that $z_{ij} = +1$ if $i = j$ (the diagonal, representing true matches) and $z_{ij} = -1$ otherwise (all off-diagonal elements, representing negatives).

The loss function incorporates two learnable parameters: a temperature $t$ (which scales the similarities to control the sharpness of the sigmoid) and a bias $b$ . The bias is crucial because in a batch of size $N$ , there is only 1 positive and $N-1$ negatives for each anchor. Without a bias, the sigmoid would be centered at 0, which is an incorrect prior for such an imbalanced dataset. Initializing $b$ to a large negative value (e.g., $-10$ ) ensures that at the start of training, the model predicts "negative" for almost all pairs, which is statistically correct.

2. Algorithm Approach

The problem requires implementing a vectorized matrix operation followed by a reduction. The core logic involves:

Constructing the Label Matrix: Create an $N \times N$ matrix where the diagonal is $+1$ and all other entries are $-1$ .
Computing the Logits: Scale the input similarity matrix by $t$ and add the bias $b$ to every element.
Applying the Sigmoid Loss: Multiply the logits element-wise by the label matrix. For positive labels ( $+1$ ), we want $\log(\sigma(x))$ . For negative labels ( $-1$ ), we want $\log(\sigma(-x))$ . This can be unified as $\log(\sigma(z_{ij} \cdot x_{ij}))$ .
Numerical Stability: Use the identity $\log(\sigma(x)) = -\text{softplus}(-x)$ to avoid numerical underflow/overflow issues that occur when computing $\log(\text{sigmoid}(x))$ directly for large $|x|$ .
Reduction: Sum all elements of the resulting loss matrix and divide by $N$ (not $N^2$ ) to get the final scalar loss.

3. Step-by-Step Strategy

PRO

Continue the full explanation

You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.

PRO

Editor locked

The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.

solution.py

Python 3.13

Test results