SigLIP 2

Why CLIP's softmax contrastive loss is a bottleneck for scaling:

CLIP's InfoNCE loss computes pairwise similarities between ALL images and ALL texts in a batch, then applies softmax normalization. For a batch of N image-text pairs:

Step 1: Similarity Matrix — compute the N x N matrix where entry $(i,j)$ = cosine similarity between image $i$ and text $j$. For N=32,768 (CLIP's batch size), this matrix has $32,768^2 = 1.07$ billion entries, each a floating-point number — roughly 4GB just for the similarity matrix.

Step 2: Softmax Normalization — for each row (image→text direction) and each column (text→image direction), compute softmax probabilities. This requires the SUM of all entries in the row/column, meaning every single pair contributes to every other pair's loss. The loss for image $i$ matching text $i$ depends on how similar image $i$ is to ALL other texts $j \neq i$.

Step 3: Distributed Training Problem — on a cluster of K GPUs, each GPU has a local batch of N/K samples. To compute the full N x N matrix, each GPU needs ALL other GPUs' embeddings. This requires TWO all-gather operations (one for image embeddings, one for text embeddings), each transferring N x D floats across the network. For N=32K and D=768: 2 x 32,768 x 768 x 4 bytes = ~200MB of cross-GPU communication PER training step.

Step 4: Memory Scaling — the N x N similarity matrix grows quadratically. At N=32K: 4GB. At N=65K: 16GB. At N=131K: 64GB — exceeding the memory of most GPUs. This hard caps the effective batch size.

Concrete impact on CLIP training:

• On 4 TPU-v4 chips (32GB each): CLIP fits batch size 2,048 per chip (8,192 total), SigLIP fits 4,096 per chip (16,384 total)
• CLIP's optimal batch size is 98K (from OpenAI's scaling experiments), but achieving this requires 48+ GPUs just for the similarity matrix
• Smaller research labs with 4-8 GPUs can only train CLIP with batch sizes 2K-8K, where softmax performance degrades significantly — the denominator has too few negatives for effective contrastive learning
• SigLIP achieves peak performance at batch size 32K — accessible on a single 8-GPU node

Sigmoid Loss reformulates contrastive learning as independent binary classification:

Instead of asking "which text best matches this image among ALL texts?" (softmax), sigmoid asks "is THIS specific image-text pair a match — yes or no?" (binary classification). Each pair is evaluated independently.

The loss function:

$\mathcal{L} = -\frac{1}{|B|} \sum_{i=1}^{N} \sum_{j=1}^{N} \log \frac{1}{1 + e^{z_{ij}(-t \cdot x_i \cdot y_j + b)}}$

Where:

• $x_i \in \mathbb{R}^D$ is the L2-normalized image embedding from the vision encoder
• $y_j \in \mathbb{R}^D$ is the L2-normalized text embedding from the text encoder
• $t$ is a learned temperature parameter (initialized at 10.0, typically converges to ~14.0)
• $b$ is a learned bias term (initialized at -10.0, typically converges to ~-10.5) — this is unique to sigmoid loss, absent in CLIP
• $z_{ij} = +1$ if $i = j$ (positive pair), $z_{ij} = -1$ if $i \neq j$ (negative pair)
• The inner product $x_i \cdot y_j$ equals cosine similarity since embeddings are L2-normalized

Why the bias term $b$ matters:

• Without $b$, the model must push ALL negative pairs to exactly zero similarity
• With $b$, the decision boundary shifts: pairs with similarity > $-b/t \approx 0.75$ are "positive", below are "negative"
• This is more natural — many image-text pairs have moderate similarity (e.g., "a dog" and an image of a park with a small dog in the background)
• The bias learns the optimal decision threshold during training

Step-by-step comparison of one training step:

Why sigmoid outperforms softmax at practical batch sizes:

• Softmax with small batch (e.g., N=4096): denominator has only 4,095 negatives → many easy negatives, weak learning signal
• Sigmoid at N=4096: each positive pair has 4,095 independent negative signals → each contributes a meaningful gradient
• At very large batch sizes (>65K): softmax catches up because it has enough negatives, but sigmoid is already at peak
• SigLIP reaches optimal performance at N=32K; CLIP needs N=98K for the same — 3x more compute per step

SigLIP 2's additional refinements:

• Text encoder uses Gemma tokenizer with 256K vocabulary (8x larger than CLIP's 32K BPE vocabulary), critical for non-Latin scripts
• Embedding dimension scales with model: 768 (Base), 1024 (Large), 1152 (So400m), 1536 (Giant)
• Sigmoid loss is the PRIMARY objective — decoder, distillation, and masked prediction are auxiliary losses added with learned weighting coefficients