Vision-Language2025

SigLIP 2

Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features

Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, Xiaohua Zhai

Read the Paper on arXiv

Paper Overview

SigLIP 2 is a family of multilingual vision-language encoders from Google DeepMind that builds upon the original SigLIP's sigmoid-based contrastive learning with a unified multi-objective training recipe that dramatically improves semantic understanding, spatial localization, and dense feature quality — all within a single training run.

Why SigLIP 2 matters — the context: CLIP (OpenAI, 2021) established contrastive vision-language pre-training: train a vision encoder and text encoder jointly so that matching image-text pairs have high cosine similarity. But CLIP uses softmax contrastive loss, which requires global normalization across the entire batch — making distributed training inefficient and limiting batch sizes. SigLIP (2023) replaced this with sigmoid loss, treating each image-text pair as an independent binary classification. This eliminated the need for cross-GPU all-gather operations and enabled 2x larger batch sizes with the same memory.

SigLIP 2's four training objectives (all applied simultaneously):

Sigmoid Contrastive Loss — the foundation from SigLIP. Pairwise binary classification: "is this image-text pair a match?" Each pair evaluated independently with sigmoid activation, no global normalization needed. Learns temperature $t$ and bias $b$ during training
Decoder-Based Objectives — a lightweight autoregressive text decoder (4-layer Transformer) with three tasks: (a) holistic image captioning, (b) bounding box prediction given region description, (c) region-specific captioning given bounding box. These force the vision encoder to learn spatially-grounded features
Self-Distillation — an EMA teacher network (updated as $\theta_T = 0.999 \cdot \theta_T + 0.001 \cdot \theta_S$ ) provides stable targets. Student sees local crops, must match teacher's representation of the full image. Applied after 80% of training to prevent early instability
Masked Prediction — 50% of image patches replaced with mask tokens (student only). Student must predict teacher's features at masked positions via MSE loss. Forces each patch to contain meaningful local information rather than relying on global context

Model family (four scales):

Model	Params	Image Res	Patch Size	Embedding Dim
Base (B)	86M	224-512	16	768
Large (L)	303M	256-512	16	1024
So400m	400M	224-512	14	1152
Giant (g)	1B	256-384	16	1536

All models include NaFlex variants that handle dynamic resolutions while preserving native aspect ratios — critical for OCR, document understanding, and non-square images. The text encoder uses the Gemma tokenizer with 256K vocabulary (8x larger than CLIP's 32K), enabling strong multilingual support across 109 languages.

Key results: SigLIP 2 So400m outperforms the original SigLIP So400m by +2.6% on ImageNet zero-shot (83.8% → 86.4%), +4.1% on COCO retrieval, and +5.3% on ADE20K segmentation (when used as frozen backbone). The NaFlex variants add another +1-2% on OCR and document benchmarks without sacrificing general performance.

Chapter Roadmap

Click any topic to jump in

Sigmoid Loss

Per-pair sigmoid decouples samples from the batch, cutting memory from O(B²) to O(B) and unlocking million-sample batches.

enables richer supervision

Decoder Objectives

Captioning, detection, and referring-expression decoders add dense spatial supervision to contrastive training.

Self-Distillation

An EMA teacher on global crops guides the student on local crops — DINO-style grounding baked into CLIP.

strengthened by masking

Masked Prediction

Predicting teacher features at 50% masked patches gives MAE-like dense representations without a separate stage.

NaFlex

Native aspect-ratio patches packed into a fixed token budget preserve geometry for dense text and wide images.

scales across geometries & languages

Multilingual + Debias

109-language training with balanced cohorts lifts non-English zero-shot by 10–15 points without regressing English.

The foundational innovation of SigLIP (inherited by SigLIP 2): replacing CLIP's softmax-based contrastive loss with a sigmoid loss that treats each image-text pair as an independent binary classification problem. This eliminates the need for global normalization across GPUs, enables 2x larger batch sizes, and actually improves performance at practical batch sizes (<32K).

The Problem

Why CLIP's softmax contrastive loss is a bottleneck for scaling:

CLIP's InfoNCE loss computes pairwise similarities between ALL images and ALL texts in a batch, then applies softmax normalization. For a batch of N image-text pairs:

Step 1: Similarity Matrix — compute the N x N matrix where entry $(i,j)$ = cosine similarity between image $i$ and text $j$ . For N=32,768 (CLIP's batch size), this matrix has $32,768^2 = 1.07$ billion entries, each a floating-point number — roughly 4GB just for the similarity matrix.

Step 2: Softmax Normalization — for each row (image→text direction) and each column (text→image direction), compute softmax probabilities. This requires the SUM of all entries in the row/column, meaning every single pair contributes to every other pair's loss. The loss for image $i$ matching text $i$ depends on how similar image $i$ is to ALL other texts $j \neq i$ .

Step 3: Distributed Training Problem — on a cluster of K GPUs, each GPU has a local batch of N/K samples. To compute the full N x N matrix, each GPU needs ALL other GPUs' embeddings. This requires TWO all-gather operations (one for image embeddings, one for text embeddings), each transferring N x D floats across the network. For N=32K and D=768: 2 x 32,768 x 768 x 4 bytes = ~200MB of cross-GPU communication PER training step.

Step 4: Memory Scaling — the N x N similarity matrix grows quadratically. At N=32K: 4GB. At N=65K: 16GB. At N=131K: 64GB — exceeding the memory of most GPUs. This hard caps the effective batch size.

Concrete impact on CLIP training:

On 4 TPU-v4 chips (32GB each): CLIP fits batch size 2,048 per chip (8,192 total), SigLIP fits 4,096 per chip (16,384 total)
CLIP's optimal batch size is 98K (from OpenAI's scaling experiments), but achieving this requires 48+ GPUs just for the similarity matrix
Smaller research labs with 4-8 GPUs can only train CLIP with batch sizes 2K-8K, where softmax performance degrades significantly — the denominator has too few negatives for effective contrastive learning
SigLIP achieves peak performance at batch size 32K — accessible on a single 8-GPU node

The Solution

Sigmoid Loss reformulates contrastive learning as independent binary classification:

Instead of asking "which text best matches this image among ALL texts?" (softmax), sigmoid asks "is THIS specific image-text pair a match — yes or no?" (binary classification). Each pair is evaluated independently.

The loss function:

$\mathcal{L} = -\frac{1}{|B|} \sum_{i=1}^{N} \sum_{j=1}^{N} \log \frac{1}{1 + e^{z_{ij}(-t \cdot x_i \cdot y_j + b)}}$

Where:

$x_i \in \mathbb{R}^D$ is the L2-normalized image embedding from the vision encoder
$y_j \in \mathbb{R}^D$ is the L2-normalized text embedding from the text encoder
$t$ is a learned temperature parameter (initialized at 10.0, typically converges to ~14.0)
$b$ is a learned bias term (initialized at -10.0, typically converges to ~-10.5) — this is unique to sigmoid loss, absent in CLIP
$z_{ij} = +1$ if $i = j$ (positive pair), $z_{ij} = -1$ if $i \neq j$ (negative pair)
The inner product $x_i \cdot y_j$ equals cosine similarity since embeddings are L2-normalized

Why the bias term $b$ matters:

Without $b$ , the model must push ALL negative pairs to exactly zero similarity
With $b$ , the decision boundary shifts: pairs with similarity > $-b/t \approx 0.75$ are "positive", below are "negative"
This is more natural — many image-text pairs have moderate similarity (e.g., "a dog" and an image of a park with a small dog in the background)
The bias learns the optimal decision threshold during training

Step-by-step comparison of one training step:

Operation	CLIP (Softmax)	SigLIP (Sigmoid)
Communication	2 all-gather ops (images + texts)	1 all-gather (texts only, chunked)
Similarity matrix	Full N x N (quadratic memory)	Chunked N x N (computed in tiles)
Normalization	Global softmax per row + column	Per-pair sigmoid (no cross-pair dependency)
Memory	O(N^2) for matrix + gradients	O(N x chunk_size) — process in tiles
Batch size at 4 TPUs	~2,048/chip = 8,192 total	~4,096/chip = 16,384 total

Why sigmoid outperforms softmax at practical batch sizes:

Softmax with small batch (e.g., N=4096): denominator has only 4,095 negatives → many easy negatives, weak learning signal
Sigmoid at N=4096: each positive pair has 4,095 independent negative signals → each contributes a meaningful gradient
At very large batch sizes (>65K): softmax catches up because it has enough negatives, but sigmoid is already at peak
SigLIP reaches optimal performance at N=32K; CLIP needs N=98K for the same — 3x more compute per step

SigLIP 2's additional refinements:

Text encoder uses Gemma tokenizer with 256K vocabulary (8x larger than CLIP's 32K BPE vocabulary), critical for non-Latin scripts
Embedding dimension scales with model: 768 (Base), 1024 (Large), 1152 (So400m), 1536 (Giant)
Sigmoid loss is the PRIMARY objective — decoder, distillation, and masked prediction are auxiliary losses added with learned weighting coefficients

Key Points

Sigmoid loss: binary classification per pair with learned temperature $t$ (~~14.0) and bias $b$ (~~-10.5) — no global normalization, no cross-pair dependency

Softmax loss: requires full N x N similarity matrix with global softmax normalization per row and column — quadratic memory O(N^2) and two cross-GPU all-gather operations

Sigmoid fits 2x larger batch sizes: 4,096/chip vs 2,048/chip on TPU-v4 by processing the similarity matrix in tiles rather than materializing the full N x N matrix

Peak performance at batch size 32K (sigmoid) vs 98K (softmax) — sigmoid achieves the same accuracy with 3x less compute per training step

Bias term $b$ is unique to sigmoid: learns the optimal positive/negative decision boundary (~0.75 cosine similarity), more natural than softmax's implicit threshold

SigLIP 2 uses Gemma tokenizer with 256K vocabulary (8x CLIP's 32K) — critical for multilingual support across 109 languages, especially non-Latin scripts

Mathematical Formulation

Sigmoid Contrastive Loss

$\mathcal{L}_{sig} = -\log \sigma(z_{ij} \cdot (t \cdot x_i \cdot y_j + b))$

Each image-text pair evaluated independently. z=+1 for positive pairs (matching), z=-1 for negatives. Temperature t and bias b are LEARNED parameters — t controls sharpness of the similarity distribution, b shifts the decision boundary.

CLIP Softmax (InfoNCE) Loss

$\mathcal{L}_{soft} = -\log \frac{e^{\tau \cdot x_i \cdot y_i}}{\sum_{j=1}^{N} e^{\tau \cdot x_i \cdot y_j}}$

Loss for image i depends on similarity with ALL N texts in the batch. The denominator requires global summation — this is why cross-GPU all-gather is needed and why memory is quadratic.

Memory Comparison

$\text{Softmax: } O(N^2) \quad \text{vs} \quad \text{Sigmoid: } O(N \times C)$

N = batch size, C = chunk size for tiled processing. At N=32K: softmax needs ~4GB for similarity matrix, sigmoid processes in C=4K tiles needing ~512MB — enabling 2x larger batches.

Mathematical Intuition

CLIP's softmax normalizes over every sample in a batch, forcing $\mathcal{O}(B)$ cross-device communication. Sigmoid loss treats every image-text pair independently: $\mathcal{L} = -\sum_{i,j}\log \sigma(z_{ij}\cdot s_{ij})$ where $z_{ij}\in\{+1,-1\}$ . Memory drops from $\mathcal{O}(B^2)$ to $\mathcal{O}(B)$ per device, enabling batch sizes up to $\sim 1$ M.

Premium Content

Upgrade to PixelBank Premium to unlock this content.

Back to All Papers

Study Plans

VLM Study Plan All Papers Practice Problems