SigLIP Loss with the Learnable Bias
Problem Statement
SigLIP replaces CLIP's softmax with an independent binary sigmoid loss per image-text pair, adding a learnable temperature t and bias b that correct for the extreme positive/negative imbalance. Implement it.
Background
For a batch of N pairs with cosine similarities sim (N x N), define the label matrix z_{ij} = +1 on the diagonal (matches) and z_{ij} = -1 off it (all other pairs are negatives). The SigLIP loss is
L=−N1∑i∑jlogσ(zij(t⋅simij+b))
Note the normalization is by N (the batch size), not N^2. The bias b is initialized very negative (about -10) so training starts near the correct all-negative prior. Use log_sigmoid(x) = -softplus(-x) for stability.
Your Task
Implement:
def siglip_loss(sim, t, b):
Return the scalar loss rounded to 4 decimals.
Input Format
- sim: N x N nested list of cosine similarities.
- t (float): temperature/scale.
- b (float): bias.
Output Format
- A float rounded to 4 decimals.
Sample
print(siglip_loss([[1.0, 0.0], [0.0, 1.0]], 1.0, 0.0))
Output:
1.0064
Example:
print(siglip_loss([[1.0, 0.0], [0.0, 1.0]], 1.0, 0.0))
1.0064
Logits t*sim+b are [[1,0],[0,1]]; with signs [[+,-],[-,+]] the diagonal terms use -log-sigmoid(1)=0.3133 and the off-diagonals -log-sigmoid(0)=0.6931. The four values sum to 2.0128; divided by N=2 gives 1.0064.
Constraints:
1 <= N <= 512.- Labels are
+1on the diagonal,-1elsewhere. - Sum over all
N*Npairs but divide byN. - Use a numerically stable
log_sigmoid; round to 4 decimals.
1. Background Knowledge
Contrastive learning in vision-language models (VLMs) aligns paired image and text embeddings by maximizing similarity for matching pairs while minimizing it for non-matching pairs. The original CLIP model uses a symmetric cross-entropy loss over a softmax distribution across the entire batch, treating each row (or column) of the similarity matrix as a multi-class classification problem where the correct match is the positive class.
SigLIP (Sigmoid Loss for Language Image Pre-Training) replaces this softmax with an independent binary sigmoid for every single pair in the N×N matrix. This decouples the computation: the probability of pair (i,j) being a match is modeled independently of other pairs. The label matrix Z is defined such that zij=+1 if i=j (the diagonal, representing true matches) and zij=−1 otherwise (all off-diagonal elements, representing negatives).
The loss function incorporates two learnable parameters: a temperature t (which scales the similarities to control the sharpness of the sigmoid) and a bias b. The bias is crucial because in a batch of size N, there is only 1 positive and N−1 negatives for each anchor. Without a bias, the sigmoid would be centered at 0, which is an incorrect prior for such an imbalanced dataset. Initializing b to a large negative value (e.g., −10) ensures that at the start of training, the model predicts "negative" for almost all pairs, which is statistically correct.
2. Algorithm Approach
The problem requires implementing a vectorized matrix operation followed by a reduction. The core logic involves:
- Constructing the Label Matrix: Create an N×N matrix where the diagonal is +1 and all other entries are −1.
- Computing the Logits: Scale the input similarity matrix by t and add the bias b to every element.
- Applying the Sigmoid Loss: Multiply the logits element-wise by the label matrix. For positive labels (+1), we want log(σ(x)). For negative labels (−1), we want log(σ(−x)). This can be unified as log(σ(zij⋅xij)).
- Numerical Stability: Use the identity log(σ(x))=−softplus(−x) to avoid numerical underflow/overflow issues that occur when computing log(sigmoid(x)) directly for large ∣x∣.
- Reduction: Sum all elements of the resulting loss matrix and divide by N (not N2) to get the final scalar loss.
3. Step-by-Step Strategy
Continue the full explanation
You're reading the free preview. Unlock the complete walkthrough, the code editor, test runner and reference solution with Premium.
Editor locked
The code editor is locked for Pro problems. It is only available for free problems. Please upgrade to gain access to the code editor for all problems.