Chapter 2: Vision Encoders

Week 3-4

Explore the vision encoders that power modern VLMs: from convolutional feature extractors to Vision Transformers, CLIP, SigLIP, DINOv2, and scaled architectures like InternViT. Understand how each encodes images into the feature representations that language models reason over.

Topics: 7
Demo: 1
Videos: 3
Min read: 48

Chapter Overview

The vision encoder is the first component in any VLM pipeline. Its job is to transform raw pixel data---a dense 3D tensor of RGB values---into a sequence of high-dimensional feature vectors that capture the semantic content of the image. The quality of these visual features determines the upper bound of what the entire VLM can perceive and reason about.

The evolution of vision encoders mirrors the broader history of computer vision:

Convolutional Neural Networks (CNNs): ResNet, EfficientNet, and their variants dominated from 2015 to 2020. They produce hierarchical feature maps through learned convolutional filters, capturing increasingly abstract patterns at each layer.
Vision Transformers (ViT): Dosovitskiy et al. (2020) showed that the same transformer architecture used for language could process images by treating image patches as tokens. ViTs have largely replaced CNNs as the standard vision backbone.
Contrastively-trained encoders: CLIP (2021) and SigLIP (2023) train ViTs to produce features aligned with text embeddings, creating a shared vision-language representation space. These are the most common encoders in modern VLMs.
Self-supervised encoders: DINOv2 (2023) trains ViTs without any text supervision, producing rich visual features through self-distillation. These features excel at dense prediction tasks.
Scaled encoders: InternViT-6B (2024) pushes the parameter count to 6 billion, demonstrating that scaling vision encoders improves VLM performance, analogous to scaling laws for LLMs.

The key trade-offs in choosing a vision encoder are:

Feature quality: How much semantic information is captured per patch token?
Resolution handling: Can the encoder process high-resolution images without quadratic cost explosion?
Alignment to language: Are the features already in a space compatible with text representations?
Computational cost: How many FLOPs per image, and how many output tokens?

$\text{Visual Tokens} = \text{ViT}(\text{Patchify}(\mathbf{I})) \in \mathbb{R}^{N \times d}$

where $N = (H / p) \times (W / p)$ is the number of patches for image resolution $H \times W$ and patch size $p$ .

Chapter Roadmap

Click any topic to jump in

Explainer videoInteractive demo

CNN Feature Extractors

How convolutional networks extract hierarchical visual features — the starting point before transformers.

CNNs as Vision BackbonesLimitations of CNNs for VLMs

From convolutions to attention

Vision Transformer (ViT)

Splitting images into patches and processing them with self-attention — the architecture that replaced CNNs for VLMs.

Patch EmbeddingSelf-Attention on Image PatchesCLS Token and Feature AggregationPositional Encoding Strategies

Three training paradigms for ViTs

Contrastive, sigmoid contrastive, and self-supervised

CLIP Encoder

Contrastive pre-training aligns vision features with language — the most common VLM backbone.

Contrastive Pre-Training ObjectiveCLIP ViT-L/14: The Standard BackboneWhy CLIP Features Work for VLMs

SigLIP

Sigmoid loss removes the need for global batch normalization — simpler and more scalable than CLIP.

Sigmoid Contrastive LossSigLIP-SO400M ArchitectureSigLIP vs. CLIP: When to Choose Which

DINOv2

Self-distillation without labels produces rich spatial features — complementary to contrastive encoders.

Self-Distillation FrameworkDINOv2 Feature Properties

Scaling to billions of parameters

InternViT & Scaling

Scaling vision encoders to 6B parameters with dynamic resolution — pushing the frontier of visual understanding.

InternViT-6B ArchitectureDynamic Resolution and Aspect Ratio HandlingScaling Laws for Vision Encoders

Practical decision framework

Choosing an Encoder

Frozen vs fine-tuned, resolution vs tokens, feature granularity — how to pick the right encoder for your VLM.

Frozen vs. Fine-Tuned Vision EncodersResolution vs. Token Count Trade-OffFeature Granularity Comparison

Topics

7 topics in this chapter

An LLM consumes a sequence of vectors, but a photograph arrives as a dense grid of RGB intensities with no notion of a token, and nothing in the language model knows what to do with it. The chapter introduction framed the vision encoder as the component that closes that gap, and for roughly five years the only serious answer was a convolutional network. Before you can judge the Vision Transformer that follows, you need a concrete picture of what it displaced: how stacked convolutions build a hierarchy from edges to textures to whole objects, what a feature map of shape $(C, H', W')$ actually contains, and how that tensor gets flattened into something an LLM can read. The order here is deliberate. First the mechanics of CNN backbones and their output shapes, then the specific limitations that pushed the field elsewhere: local receptive fields, fixed resolution, tokens as an afterthought, no text alignment, and spatial precision traded away. This is not archaeology. Region-based CNN features powered the first generation of VQA systems, and hybrid designs still borrow convolutional stems today.

In this topic

1CNNs as Vision Backbones
2Limitations of CNNs for VLMs

1 of 2

CNNs as Vision Backbones

A CNN processes an image through a series of convolutional layers, each applying learnable filters that detect patterns at increasing levels of abstraction:

Early layers: Detect edges, textures, and color gradients (local patterns)
Middle layers: Detect parts of objects (eyes, wheels, windows)
Deep layers: Detect entire objects and scenes (faces, cars, rooms)

The output of a CNN backbone is a feature map: a 3D tensor of shape $(C, H', W')$ where $C$ is the number of feature channels, and $H', W'$ are the spatially downsampled dimensions. For ResNet-50 processing a $224 \times 224$ image, the final feature map is $(2048, 7, 7)$ ---2048 channels at $7 \times 7$ spatial resolution.

For VLMs, this feature map can be flattened into a sequence of $7 \times 7 = 49$ tokens of dimension 2048, then projected into the LLM's embedding space.

Mathematical Intuition

The feature map dimension follows $C_{\ell} \propto 2^{\ell}$ while spatial extent shrinks as $H_{\ell} = H_0 / 2^{\ell}$ . The total information budget per layer stays roughly constant: $C_{\ell} \cdot H_{\ell}^2 \approx C_0 \cdot H_0^2$ . This means no information is created or destroyed — it is merely re-encoded from spatial to channel dimensions.

Example:

A ResNet-50 backbone produces a feature map of shape (2048, 7, 7) for a 224x224 input image. How does this compare to a ViT-L/14 processing the same image?

2 of 2

Limitations of CNNs for VLMs

While CNNs remain useful in some contexts, they have fundamental limitations as VLM encoders:

Limited receptive field: Each CNN feature only captures a local region. Global context requires very deep stacking of layers, making it indirect.
Fixed spatial resolution: The feature map resolution is determined by the architecture. Changing input resolution requires interpolation or architectural changes.
No built-in token sequence: LLMs expect a sequence of tokens. Converting CNN feature maps to tokens is an afterthought rather than a native design.
No text alignment: CNNs are typically pre-trained on ImageNet classification, which provides category-level understanding but not the rich semantic features learned through text-aligned training (CLIP).
Translation equivariance, not invariance: Convolutions detect patterns regardless of position, which is useful for classification but means spatial information is partially lost---problematic for grounding tasks.

Mathematical Intuition

The effective receptive field (ERF) of a CNN grows as $O(\sqrt{L})$ with depth $L$ , not $O(L)$ as the theoretical receptive field suggests. After 50 layers of $3 \times 3$ convolutions the ERF covers roughly $\frac{1}{4}$ of the image, meaning deep CNN features are still largely local. Self-attention achieves a global receptive field ( $N \times N$ attention matrix) in a single layer.

Example:

Why did Bottom-Up Top-Down Attention (Anderson et al., 2018)---one of the most successful early VLM approaches---use Faster R-CNN instead of a standard ResNet classifier?

Theory Exercise

Problem:

Explain why Vision Transformers replaced CNNs as the dominant vision encoder in VLMs despite CNNs having lower computational cost and decades of optimization. Identify at least three architectural properties of ViTs that make them inherently better suited for the VLM pipeline.

Hints:

Consider how self-attention creates global receptive fields vs. local convolutions
Think about how ViT outputs naturally map to the token-based interface of LLMs
Consider the alignment between ViT training objectives (CLIP) and VLM requirements

Papers

Deep Residual Learning for Image Recognition (ResNet)

Blogs

CNN Explainer — Interactive Visualization

In this topic

1Patch Embedding
2Self-Attention on Image Patches
3CLS Token and Feature Aggregation
4Positional Encoding Strategies

1 of 4

Patch Embedding

$\mathbf{z}_0 = [\mathbf{x}_{\text{cls}}; \, \mathbf{x}_p^1 \mathbf{E}; \, \mathbf{x}_p^2 \mathbf{E}; \, \ldots; \, \mathbf{x}_p^N \mathbf{E}] + \mathbf{E}_{\text{pos}}$

ViT converts an image into a sequence of tokens through three steps:

Patchification: The image $\mathbf{I} \in \mathbb{R}^{H \times W \times C}$ is divided into $N = HW/p^2$ non-overlapping patches of size $p \times p$ .
Linear projection: Each patch $\mathbf{x}_p^i \in \mathbb{R}^{p^2 \cdot C}$ is flattened and multiplied by a learnable embedding matrix $\mathbf{E} \in \mathbb{R}^{(p^2 \cdot C) \times d}$ to produce a $d$ -dimensional token.
Positional encoding: A learnable positional embedding $\mathbf{E}_{\text{pos}} \in \mathbb{R}^{(N+1) \times d}$ is added to encode spatial location. A special $\mathbf{x}_{\text{cls}}$ token is prepended for global representation.

The result is a sequence of $N + 1$ tokens, each of dimension $d$ , that the transformer layers process with self-attention.

Resolution scaling: For a $p=14$ ViT, a $224 \times 224$ image produces $16 \times 16 = 256$ tokens, while a $448 \times 448$ image produces $32 \times 32 = 1024$ tokens. This quadratic scaling in token count is a key consideration for VLMs.

Mathematical Intuition

The embedding matrix $\mathbf{E} \in \mathbb{R}^{(p^2 C) \times d}$ is equivalent to a convolution with kernel size $=$ stride $= p$ . In frequency terms, the patch projection acts as a low-pass filter with cutoff wavelength $\lambda_{\min} = 2p$ pixels (Nyquist limit). Halving the patch size doubles the spatial bandwidth and quadruples the token count: $N = (H/p)^2$ .

Example:

For ViT-B/16 (patch size 16, embedding dim 768) processing a 384x384 image: (a) How many patch tokens? (b) What is the embedding matrix size? (c) Total embedding parameters?

2 of 4

Self-Attention on Image Patches

$\text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}\left(\frac{\mathbf{Q} \mathbf{K}^\top}{\sqrt{d_k}}\right) \mathbf{V}$

After patch embedding, the token sequence passes through $L$ transformer layers, each consisting of multi-head self-attention (MHSA) and a feed-forward network (FFN):

$\mathbf{z}'_l = \text{MHSA}(\text{LN}(\mathbf{z}_{l-1})) + \mathbf{z}_{l-1}$ $\mathbf{z}_l = \text{FFN}(\text{LN}(\mathbf{z}'_l)) + \mathbf{z}'_l$

In self-attention, each patch token computes queries ( $\mathbf{Q}$ ), keys ( $\mathbf{K}$ ), and values ( $\mathbf{V}$ ) through linear projections. The attention score between patches $i$ and $j$ measures their relevance: $\alpha_{ij} = \text{softmax}(\mathbf{q}_i \cdot \mathbf{k}_j / \sqrt{d_k})$ . Each patch's output is a weighted sum of all patches' values, with weights determined by attention scores.

With $h$ attention heads, each head can learn different types of relationships (e.g., spatial proximity, color similarity, semantic similarity), and their outputs are concatenated.

Computational cost: Self-attention scales as $O(N^2 \cdot d)$ where $N$ is the number of patches. For a 224px image with 256 patches, this is manageable. For a 1344px image with 9,216 patches, the cost becomes $\sim 85M$ attention scores per layer---a significant bottleneck.

Mathematical Intuition

Attention complexity is $O(N^2 d)$ where $N = (H/p)^2$ . Doubling resolution at fixed $p$ quadruples $N$ , making attention cost scale as the fourth power of resolution: $O(H^4 / p^4)$ . This quartic scaling is the fundamental reason VLMs must use token compression or tiling for high-resolution inputs.

Example:

ViT-L has 24 transformer layers with 16 attention heads and d=1024. For an input of 576 tokens (384x384 image, patch size 14), how many total attention computations are performed?

3 of 4

CLS Token and Feature Aggregation

ViTs offer two strategies for producing a global image representation:

CLS token: A special learnable token prepended to the patch sequence. Through self-attention, it aggregates information from all patches across all layers. The final CLS token's hidden state serves as a global image embedding.
Global Average Pooling (GAP): Average all patch token outputs to produce a single vector. Eliminates the need for a special token.

For VLMs, neither approach is typically used in isolation. Instead, VLMs use all patch tokens as visual input to the language model, because the LLM needs fine-grained spatial information to answer questions like "What color is the object on the left?" The CLS token may be included as an additional summary token, but the patch tokens are essential.

Some VLMs (BLIP-2, Flamingo) use only a subset of patch tokens or compress them through a bridge module to reduce the sequence length for the LLM.

Mathematical Intuition

The CLS token acts as a rank-1 bottleneck: it projects the full $N \times d$ patch matrix into a single $d$ -dimensional vector, discarding at least $(N-1) \cdot d$ degrees of freedom. For $N = 256$ and $d = 1024$ , this means retaining at most $\frac{1}{256} \approx 0.4\%$ of the spatial information, which is why VLMs prefer the full patch sequence.

Example:

Why do VLMs typically pass ALL patch tokens to the LLM rather than just the CLS token? Doesn't the CLS token already contain a summary of the entire image?

4 of 4

Positional Encoding Strategies

$\mathbf{E}_{\text{pos}}^{(i)} \in \mathbb{R}^d, \quad i \in \{0, 1, \ldots, N\}$

Since self-attention is permutation-invariant (it treats the input as a set, not a sequence), positional information must be explicitly injected. ViT uses learnable positional embeddings---a matrix $\mathbf{E}_{\text{pos}} \in \mathbb{R}^{(N+1) \times d}$ that is added to the token embeddings.

Types of positional encoding in vision encoders:

1D learnable (original ViT): Each of the $N+1$ positions gets a unique learned vector. The model must learn 2D spatial structure from data.
2D sinusoidal: Separate sine/cosine encodings for row and column positions, concatenated. Encodes 2D structure explicitly.
Rotary Position Embedding (RoPE): Applied to queries and keys in attention, encodes relative distances. Used in newer vision encoders for better resolution generalization.

Resolution transfer: When fine-tuning at a different resolution than pre-training, the number of patches changes. Positional embeddings are typically interpolated (bicubic) to the new grid size. This works well in practice but introduces a small performance drop at very different resolutions.

Mathematical Intuition

Learned 1D positional embeddings implicitly encode a 2D grid: the cosine similarity between positions $(r_1, c_1)$ and $(r_2, c_2)$ decays with Manhattan distance. Bicubic interpolation to a new grid works because the learned embeddings are smooth in 2D — nearby positions have similar vectors, so intermediate values are well-approximated by polynomial interpolation.

Example:

A ViT is pre-trained at 224x224 resolution (16x16 = 256 patches with p=14). You want to use it at 448x448 (32x32 = 1024 patches). How do you handle the positional embeddings?

Theory Exercise

Problem:

ViT-L/14 produces 256 patch tokens for a 224x224 image. A user submits a 1024x1024 image. Calculate: (a) the number of tokens, (b) the self-attention FLOPs relative to 224x224, and (c) propose two strategies to handle this without exploding compute.

Hints:

Token count scales quadratically with image dimension when patch size is fixed
Self-attention FLOPs scale as O(N^2), so the total scaling is quartic in image dimension
Think about both pre-processing (resize, tile) and architectural (token merging, windowed attention) solutions

Papers

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Blogs

Vision Transformers (ViT) — Hugging Face

In this topic

1Contrastive Pre-Training Objective
2CLIP ViT-L/14: The Standard Backbone
3Why CLIP Features Work for VLMs

1 of 3

Contrastive Pre-Training Objective

$\mathcal{L}_i^{\text{img}} = -\log \frac{\exp(\text{sim}(\mathbf{v}_i, \mathbf{t}_i) / \tau)}{\sum_{j=1}^{B} \exp(\text{sim}(\mathbf{v}_i, \mathbf{t}_j) / \tau)}$

CLIP trains a vision encoder (ViT) and a text encoder (Transformer) jointly on image-text pairs. For a batch of $B$ pairs, the training objective maximizes the cosine similarity between matching image-text pairs while minimizing it for non-matching pairs.

The loss is symmetric: $\mathcal{L} = \frac{1}{2}(\mathcal{L}^{\text{img}} + \mathcal{L}^{\text{txt}})$ , where $\mathcal{L}^{\text{img}}$ classifies images to texts and $\mathcal{L}^{\text{txt}}$ classifies texts to images.

The temperature parameter $\tau$ (learned, typically initialized to 0.07) controls the sharpness of the distribution. Lower $\tau$ makes the loss more sensitive to small similarity differences.

Training scale: CLIP was trained on 400M image-text pairs (WebImageText dataset) for 32 epochs with batch size 32,768. The large batch size is critical because each batch provides $B-1$ negative examples per sample.

Mathematical Intuition

Cosine similarity constrains embeddings to the unit hypersphere $S^{d-1}$ . The capacity grows exponentially with dimension: in $d = 512$ , roughly $e^{d/2}$ nearly-orthogonal vectors can coexist. This means CLIP can encode an astronomically large number of distinct concepts, limited not by geometry but by the quality of training data.

Example:

Why does CLIP need such large batch sizes (32,768)? What happens with a batch size of 32?

2 of 3

CLIP ViT-L/14: The Standard Backbone

CLIP ViT-L/14 has become the de facto standard vision encoder for open-source VLMs. Its specific configuration:

Parameter	Value
Architecture	ViT-Large
Patch size	14x14
Hidden dim	1024
Layers	24
Heads	16
Parameters	304M
Training data	400M image-text pairs
Resolution	224x224 (336x336 with fine-tuning)

Feature properties:

Output: 257 tokens (1 CLS + 256 patches) at 224x224
CLS token: Global image representation, used for retrieval
Patch tokens: Local features, used by VLMs for detailed reasoning
Features are normalized (unit vectors on the hypersphere)
Features are aligned with CLIP text encoder's output space

Used by: LLaVA, LLaVA-NeXT, ShareGPT4V, Cambrian, and dozens of other VLMs.

Mathematical Intuition

At 336px, token count is $(336/14)^2 = 576$ versus $(224/14)^2 = 256$ at 224px — a $2.25\times$ increase. The LLM attention cost scales quadratically in sequence length, so the total VLM cost increase is closer to $2.25^2 \approx 5\times$ , not $2.25\times$ . This quadratic penalty is why resolution upgrades have diminishing returns for fixed compute.

Example:

LLaVA-1.5 uses CLIP ViT-L/14 at 336x336 resolution (up from 224x224). How does this change affect the visual features and VLM performance?

3 of 3

Why CLIP Features Work for VLMs

CLIP's features are particularly effective for VLMs because of three key properties:

Language alignment: CLIP features live in the same embedding space as text. An image of a dog and the text "a dog" produce similar vectors. This means the bridge module only needs to handle dimensional transformation, not conceptual alignment.
Semantic richness: Trained on 400M diverse image-text pairs (not just ImageNet's 1,000 categories), CLIP features encode concepts far beyond object categories: actions, attributes, scenes, text in images, emotions, styles, and abstract concepts.
Compositional understanding: Because training captions describe compositions ("a red car next to a blue house"), CLIP learns to encode visual compositions, not just individual objects. This compositional understanding transfers to VLMs.

Limitation: CLIP's contrastive objective optimizes for global image-text matching, which may not fully preserve fine-grained spatial information. Patch-level features are useful but not as spatially precise as features from models trained with dense objectives (like DINOv2).

Mathematical Intuition

CLIP's contrastive loss maximizes $I(V; T)$ — the mutual information between vision and text representations — across the training set. This means CLIP features retain exactly the visual information that co-varies with language, discarding texture noise and imaging artifacts. The bridge module's job reduces to a dimensionality transform $\mathbb{R}^{1024} \to \mathbb{R}^{4096}$ rather than a full semantic realignment.

Example:

A VLM built on CLIP ViT-L can describe images fluently but struggles with precise counting ('How many apples are there?'). Why might CLIP features be suboptimal for counting, and what could help?

Theory Exercise

Problem:

CLIP was trained with symmetric contrastive loss (both image-to-text and text-to-image). Some researchers have proposed asymmetric variants where the image encoder receives stronger gradients than the text encoder. Analyze: (1) Why might asymmetric training help VLMs? (2) What could go wrong? (3) Design an experiment to test this hypothesis.

Hints:

Think about which encoder is more important for the downstream VLM task
Consider that the text encoder is often discarded (only the vision encoder is used in VLMs)
Think about how gradient flow affects what the vision encoder learns to represent

Papers

Learning Transferable Visual Models From Natural Language Supervision (CLIP)

Blogs

CLIP: Connecting Text and Images — OpenAI

In this topic

1Sigmoid Contrastive Loss
2SigLIP-SO400M Architecture
3SigLIP vs. CLIP: When to Choose Which

1 of 3

Sigmoid Contrastive Loss

$\mathcal{L} = -\frac{1}{B^2} \sum_{i=1}^{B} \sum_{j=1}^{B} \left[ y_{ij} \log \sigma(z_{ij}) + (1 - y_{ij}) \log (1 - \sigma(z_{ij})) \right]$

where $z_{ij} = (\mathbf{v}_i \cdot \mathbf{t}_j - b) \cdot e^t$ , $y_{ij} = \mathbb{1}[i = j]$ indicates matching pairs, and $\sigma$ is the sigmoid function.

CLIP's softmax loss treats each image as an $N$ -way classification problem: "which of the $N$ texts matches this image?" SigLIP instead treats each image-text pair independently: "does this specific image match this specific text?" This is a binary classification problem for each of the $B^2$ pairs in the batch.

Key advantages over CLIP's softmax:

No global normalization: Softmax requires computing scores across the entire batch (global communication). Sigmoid computes each pair independently, enabling efficient distributed training.
Better batch scaling: CLIP's softmax benefits from very large batches (32K) because more negatives = harder classification. SigLIP's sigmoid works well with smaller batches because each pair provides an independent learning signal.
Smoother gradients: Softmax can produce very sharp gradients for hard negatives, causing training instability. Sigmoid provides more uniform gradients.
Learnable bias: The bias term $b$ and temperature $t$ are learnable, allowing the model to calibrate its confidence.

Mathematical Intuition

The softmax denominator $\sum_j e^{z_{ij}}$ couples all gradients in a row — changing any single score $z_{ij}$ changes all $B$ probabilities via the partition function. The sigmoid $\sigma(z_{ij})$ decouples pairs completely: $\partial \mathcal{L} / \partial z_{ij}$ depends only on $(z_{ij}, y_{ij})$ . This makes SigLIP embarrassingly parallel across GPUs, eliminating the all-reduce bottleneck that limits CLIP's distributed scaling.

Example:

For a batch of B=4 image-text pairs, CLIP computes a 4x4 softmax loss (4+4=8 cross-entropy terms). SigLIP computes 4x4=16 binary cross-entropy terms. How does the gradient computation differ?

2 of 3

SigLIP-SO400M Architecture

SigLIP-SO400M (Shape-Optimized, 400M parameters) is the most commonly used SigLIP variant in modern VLMs. It was trained on WebLI (a curated web-scale dataset) with the sigmoid loss.

Key specifications:

Vision encoder: ViT with ~400M parameters
Patch size: 14
Hidden dimension: 1152
Layers: 27
Native resolution: 384x384 (producing 729 tokens)

Used in:

PaliGemma (Google)
LLaVA-OneVision
InternVL (as one of the encoder options)
Idefics2 (HuggingFace)

SigLIP-SO400M consistently outperforms CLIP ViT-L/14 (304M) on VLM benchmarks, likely due to both the sigmoid loss and the larger model size (400M vs. 304M parameters).

Mathematical Intuition

The hidden dimension increase from $1024$ (CLIP) to $1152$ (SigLIP-SO400M) gives $1152^2 / 1024^2 \approx 1.27\times$ more parameters per attention layer. Combined with 27 vs 24 layers, the total encoder has $\sim 1.32\times$ more capacity per layer and $\sim 1.48\times$ more total depth-capacity, explaining the consistent +1-3% benchmark improvements.

Example:

SigLIP-SO400M has 1152-dim embeddings and 729 tokens at 384x384. A VLM using Llama-3-8B (4096-dim) needs to bridge this. Calculate the bridge dimensions and compare to CLIP ViT-L.

3 of 3

SigLIP vs. CLIP: When to Choose Which

The choice between SigLIP and CLIP as a VLM vision encoder depends on several factors:

Choose SigLIP when:

You need a larger vision encoder (SO400M > ViT-L/304M)
Training your own CLIP-like model at scale (sigmoid loss is easier to distribute)
Building a VLM that will be fine-tuned end-to-end (smoother gradients)

Choose CLIP when:

Using an existing CLIP-based pipeline (LLaVA, etc.)
You need the extensive CLIP ecosystem (pre-computed features, fine-tuned variants)
Model size is constrained (CLIP ViT-B/16 at 86M is much smaller than SigLIP-SO400M)

In practice:

Both produce features of similar quality for VLMs
SigLIP has a slight edge on most benchmarks (+1-3%)
The specific VLM training recipe (data, bridge design, LLM) matters more than the vision encoder choice
Modern VLMs increasingly default to SigLIP for new projects

Mathematical Intuition

The gradient variance of softmax loss scales as $O(1/B)$ (variance reduction through averaging over $B$ negatives), while sigmoid loss variance scales as $O(1/B^2)$ (averaging over $B^2$ pairs). For $B = 1024$ , sigmoid has $1024\times$ lower gradient variance, directly translating to more stable optimization and less sensitivity to learning rate selection.

Example:

A research lab wants to train a CLIP-like model on 2 billion proprietary medical image-text pairs distributed across 256 GPUs. Should they use softmax (CLIP) or sigmoid (SigLIP) loss?

Theory Exercise

Problem:

SigLIP's loss function treats each of the B^2 pairs in a batch independently. This means there are B matching pairs and B^2 - B non-matching pairs. For B=1024, the ratio is 1024:1047552 (about 1:1023). Analyze: (1) Does this class imbalance affect training? (2) How does it compare to CLIP's implicit handling of negatives? (3) Propose a modification to address any issues.

Hints:

Think about the gradient magnitude from positive vs. negative pairs
Consider how CLIP's softmax naturally handles the ratio via normalization
Think about focal loss or other techniques for handling class imbalance

Papers

Sigmoid Loss for Language Image Pre-Training (SigLIP)

Blogs

SigLIP — Hugging Face

In this topic

1InternViT-6B Architecture
2Dynamic Resolution and Aspect Ratio Handling
3Scaling Laws for Vision Encoders

1 of 3

InternViT-6B Architecture

InternViT-6B is a Vision Transformer with 6 billion parameters, trained with both contrastive (CLIP-like) and generative objectives. Its scale dwarfs conventional vision encoders:

Model	Params	Layers	Hidden Dim	Heads
CLIP ViT-L/14	304M	24	1024	16
SigLIP-SO400M	400M	27	1152	16
InternViT-6B	5,902M	48	3200	25

InternViT-6B has nearly 20x more parameters than CLIP ViT-L. This scale enables:

Richer per-patch features (3200-dim vs. 1024-dim)
More transformer layers for deeper contextual processing
Better fine-grained understanding (OCR, small objects, dense scenes)

Training: Progressive training at multiple resolutions (224 $\rightarrow$ 384 $\rightarrow$ 448), with a custom dynamic resolution strategy that handles arbitrary aspect ratios.

Mathematical Intuition

With 48 layers and $d = 3200$ , the total self-attention FLOP count per image at 448px is $48 \times 2 \times 1024^2 \times 3200 \approx 3.2 \times 10^{11}$ — about $20\times$ that of CLIP ViT-L. The per-head dimension of $128 = 3200/25$ matches the value used in Llama-2-70B, suggesting a converged optimal head size across modalities.

Example:

InternViT-6B uses 3200-dim hidden size with 25 attention heads. What is the per-head dimension, and how does this compare to standard ViTs?

2 of 3

Dynamic Resolution and Aspect Ratio Handling

Real-world images come in diverse resolutions and aspect ratios. A naive approach (resize to fixed square) distorts content and loses information. Modern scaled vision encoders handle this through dynamic resolution strategies:

Padding: Resize the shorter side to the target, pad the longer side with zeros. Preserves aspect ratio but wastes computation on padding.
Sub-image tiling (InternVL, LLaVA-NeXT): Split the image into tiles that fit the encoder's native resolution. A 1280x720 image might become 3 tiles of 448x448 (overlapping) plus a 448x448 downsampled thumbnail.
NaFlex (SigLIP2): Dynamically adjust the number of patches based on image content, producing a variable-length token sequence.

Why this matters for VLMs: Document images, screenshots, and photographs have very different aspect ratios. A letter-sized document (8.5:11) contains critical text that would be illegible if squeezed into a 1:1 square.

Mathematical Intuition

For an image of height $H$ and width $W$ , the optimal $(r, c)$ tiling minimizes aspect-ratio distortion: $\arg\min_{r \cdot c \leq T} |c/r - W/H|$ . The total token count is $r \cdot c \cdot (S/p)^2$ where $S$ is the tile size, giving a linear-in-tiles (not quadratic-in-resolution) scaling of compute — the key insight that makes high-resolution processing tractable.

Example:

A user uploads a 3000x2000 pixel photograph to a VLM with InternViT-6B (max resolution 448x448 per tile, max 12 tiles). How is the image processed?

3 of 3

Scaling Laws for Vision Encoders

The success of InternViT-6B suggests that vision encoder scaling laws exist, similar to those for LLMs. Empirical observations:

More parameters = better features: InternViT-6B produces richer per-token features than ViT-L, leading to better VLM performance across all benchmarks.
Diminishing returns at fixed resolution: Scaling the encoder while keeping resolution fixed shows diminishing improvements. The bottleneck shifts to the number of patches (information available to encode).
Resolution and parameters must co-scale: The best results come from increasing both resolution (more patches) and encoder size (richer features per patch) simultaneously.
Vision encoder scaling is underexplored: Most VLM research scales the LLM (7B $\rightarrow$ 13B $\rightarrow$ 70B) while keeping the vision encoder fixed. InternVL showed that allocating more parameters to vision has outsized benefits.

Implication: The optimal VLM may have a much larger vision encoder than current designs. If LLMs benefit from 70B+ parameters, there is no reason vision encoders should stop at 6B.

Mathematical Intuition

Empirically, vision encoder performance scales as $\text{score} \sim \alpha \log(N_{\text{params}}) + \beta \log(N_{\text{tokens}})$ , where both terms contribute independently. Doubling parameters at fixed resolution gives $\alpha \log 2 \approx 0.7\alpha$ improvement, while doubling resolution (quadrupling tokens) gives $2\beta \log 2 \approx 1.4\beta$ . When $\beta > \alpha$ , resolution scaling dominates — explaining why InternVL co-scales both.

Example:

A VLM has two configurations: (A) InternViT-6B + Llama-7B = 13B total, (B) CLIP ViT-L-304M + Llama-13B = 13.3B total. Same total params. Which performs better on visual tasks?

Theory Exercise

Problem:

Design a scaling experiment to determine the optimal parameter allocation between vision encoder and LLM for a fixed total compute budget. Specify: (1) the configurations to test, (2) the benchmarks to evaluate, (3) how you would control for confounding variables, and (4) what outcome would change current VLM design practices.

Hints:

Consider at least 5 configurations with different vision:LLM parameter ratios
Include benchmarks that stress-test vision (OCR, spatial) and reasoning (MMMU) separately
Think about confounding variables: training data quality, bridge design, training recipe

Papers

InternVL: Scaling up Vision Foundation Models

Blogs

InternVL Project Page

In this topic

1Self-Distillation Framework
2DINOv2 Feature Properties

1 of 2

Self-Distillation Framework

$\mathcal{L}_{\text{DINO}} = -\sum_{x \in \{x_1^g, x_2^g\}} \sum_{x' \neq x} p_t(x')^\top \log p_s(x)$

DINOv2 uses a student-teacher framework with exponential moving average (EMA) updates:

Student network: Processes both global and local crop views of the image. Weights are updated by gradient descent.
Teacher network: Processes only global crop views. Weights are an EMA of the student: $\theta_t \leftarrow m \cdot \theta_t + (1 - m) \cdot \theta_s$ where $m \approx 0.996$ .

The loss encourages the student to produce the same output distribution as the teacher across different augmented views of the same image. Since the teacher sees global views while the student sees both global and local crops, the student learns to predict global context from local information.

No negative pairs needed: Unlike CLIP/SigLIP, DINO does not require explicit negative examples. The centering and sharpening of the teacher's outputs prevent mode collapse (all images mapping to the same representation).

Mathematical Intuition

The EMA update $\theta_t \leftarrow m \theta_t + (1 - m) \theta_s$ with $m = 0.996$ means the teacher's effective learning rate is $(1 - m) = 0.004$ times the student's. Over $K$ steps, the teacher integrates a Polyak-averaged trajectory: $\theta_t^{(K)} = (1-m) \sum_{k=0}^{K-1} m^k \theta_s^{(K-k)}$ , an exponentially-weighted moving average that smooths away gradient noise and prevents the teacher from collapsing to trivial solutions.

Example:

DINOv2 uses 2 global crops (covering 50-100% of the image) and 10 local crops (covering 5-50% of the image). Why this asymmetry, and what does each crop type contribute to learning?

2 of 2

DINOv2 Feature Properties

DINOv2 features have distinctive properties compared to CLIP:

Dense, spatially precise features: Because DINOv2 is trained with local crops, its patch-level features encode precise spatial information. This makes DINOv2 superior for dense prediction tasks (segmentation, depth estimation).
Instance-level discrimination: DINOv2 can distinguish between individual instances of the same class (e.g., different dogs) better than CLIP, which primarily learns category-level features.
No language alignment: DINOv2 features are not aligned to text. They encode visual similarity and structure but cannot be directly compared to text embeddings. This requires a more capable bridge module in VLMs.
Emergent properties: DINOv2 features exhibit remarkable emergent behaviors---attention maps naturally segment objects, patch-level features cluster semantically without supervision, and linear probes on frozen features achieve strong performance on diverse tasks.

For VLMs: DINOv2's dense features complement CLIP's semantic features. Some VLMs (Cambrian-1, Eagle) combine both encoders to get the best of both worlds.

Mathematical Intuition

DINOv2 features form clusters in $\mathbb{R}^d$ that correspond to semantic parts without any label supervision: the $k$ -nearest-neighbor graph of patch features naturally segments objects. Formally, the learned feature map $f: \mathbb{R}^{p \times p \times 3} \to \mathbb{R}^d$ induces a metric $d(i,j) = \|f(p_i) - f(p_j)\|$ where $d(i,j) < \epsilon$ iff patches $i, j$ belong to the same semantic region. This emerges because the multi-crop loss forces local patches to predict global structure.

Example:

A VLM uses CLIP ViT-L for visual encoding. The model can describe images well but struggles with: (a) counting objects, (b) identifying specific parts of objects, (c) depth ordering. Would adding DINOv2 features help?

Theory Exercise

Problem:

DINOv2 is trained without any text supervision, yet when used in VLMs, the bridge module can learn to map DINOv2 features to the LLM's embedding space. Why does this work? What must the bridge module learn that is different from when CLIP features are used?

Hints:

Think about what information is already present in DINOv2 features vs. what is missing
Consider how CLIP's text alignment makes the bridge module's job easier
Think about the bridge module's role as a 'translator' between representation spaces

Papers

DINOv2: Learning Robust Visual Features without Supervision

Blogs

DINOv2 — Meta AI

In this topic

1Frozen vs. Fine-Tuned Vision Encoders
2Resolution vs. Token Count Trade-Off
3Feature Granularity Comparison

1 of 3

Frozen vs. Fine-Tuned Vision Encoders

A critical design choice: should the vision encoder's weights be fixed (frozen) during VLM training, or updated (fine-tuned)?

Frozen encoder:

Preserves the pre-trained feature quality
Much cheaper to train (no gradients through the encoder)
Allows using pre-computed features (offline extraction)
Used by: BLIP-2, Flamingo, early LLaVA

Fine-tuned encoder:

Features adapt specifically to the VLM's needs
Can learn task-specific visual patterns not captured during pre-training
Risk: catastrophic forgetting of pre-trained capabilities
Used by: LLaVA-1.5+, InternVL, Qwen-VL

Practical guideline: Start with a frozen encoder (faster iteration). If performance plateaus, unlock the encoder with a low learning rate ( $10\times$ lower than the bridge module).

Mathematical Intuition

Fine-tuning a pre-trained encoder is equivalent to walking along the loss landscape from the CLIP/SigLIP minimum toward the VLM-specific minimum. If the two minima are in the same basin (small distributional shift), a small learning rate suffices. If they are in different basins (large shift), aggressive fine-tuning causes catastrophic forgetting — the weights cross a loss barrier and lose the pre-trained basin entirely. LoRA constrains updates to a low-rank subspace $\Delta W = BA$ with $\text{rank}(BA) \ll d$ , keeping the trajectory close to the original minimum.

Example:

A VLM with a frozen CLIP encoder achieves 75% on MMBench. After unfreezing the vision encoder (lr=1e-6), it achieves 79% after 1 epoch but drops to 71% after 5 epochs. What happened?

2 of 3

Resolution vs. Token Count Trade-Off

Higher resolution means more visual detail but also more tokens, increasing computational cost quadratically in the LLM's attention layers.

Resolution	Patches (p=14)	LLM Attention Cost
224x224	256	1.0x (baseline)
336x336	576	5.1x
448x448	1,024	16.0x
672x672	2,304	81.0x
1344x1344	9,216	1,296x

When high resolution matters:

OCR / document understanding (text must be readable)
Fine-grained recognition (bird species, product details)
Spatial reasoning with small objects
Chart/graph interpretation

When 224-336px is sufficient:

Scene classification
Object detection (coarse)
General image captioning
Binary VQA (yes/no questions)

Mathematical Intuition

LLM self-attention over $N$ visual tokens costs $O(N^2 d)$ FLOPs per layer. Since $N = (H/p)^2$ , the total cost is $O(H^4 / p^4)$ — quartic in resolution. Going from 224px to 672px ( $3\times$ ) multiplies attention cost by $3^4 = 81\times$ . Token-compressing bridges break this quartic scaling by mapping $N$ tokens to $M \ll N$ , reducing cost to $O(M^2 d)$ .

Example:

A VLM must process 100 images per minute for a real-time surveillance system. Each image is 1920x1080. The LLM can process 10,000 tokens/second. What resolution should the vision encoder use?

3 of 3

Feature Granularity Comparison

Different encoders produce features with different granularity characteristics:

CLIP ViT-L/14:

Strong semantic features (category-level)
Good text alignment
Weaker spatial precision
Best for: general VQA, captioning, retrieval

SigLIP-SO400M:

Slightly better semantic features than CLIP
Efficient training
Better calibrated confidence
Best for: general-purpose VLMs, production deployment

DINOv2 ViT-g:

Excellent spatial features (instance-level)
No text alignment (needs stronger bridge)
Best for: dense prediction, counting, segmentation, grounding

InternViT-6B:

Richest overall features (most parameters)
Strong on both semantics and spatial detail
Highest computational cost
Best for: document understanding, maximum quality regardless of cost

Combined encoders (Cambrian-1 approach):

CLIP + DINOv2 features concatenated per patch
Best of both worlds: semantic + spatial
Double the bridge module size and visual tokens
Best for: research, benchmarks, when accuracy is paramount

Mathematical Intuition

The information content per token scales roughly as $\log_2(\text{params}/N)$ bits, where $\text{params}$ is the encoder size and $N$ is the token count. CLIP ViT-L gives $\log_2(304M/256) \approx 20$ bits/token while InternViT-6B gives $\log_2(5902M/256) \approx 24.5$ bits/token. These extra 4.5 bits per token compound across 256 tokens, yielding $\sim 1152$ additional bits of visual information — enough to encode roughly 150 extra characters of text-equivalent detail.

Example:

You are building three VLMs for different products. Select the optimal vision encoder for each: (1) A mobile photography assistant, (2) A document digitization service, (3) A visual search engine.

Theory Exercise

Problem:

A VLM research team has budget for one major training run. They must decide between: (A) CLIP ViT-L at 672x672 (2,304 tokens per image), or (B) InternViT-6B at 224x224 (256 tokens per image). Both use the same LLM. Analyze which configuration will perform better on (1) OCR tasks, (2) visual reasoning tasks, and (3) general VQA. Consider both feature quality and computational efficiency.

Hints:

More tokens at moderate feature quality vs. fewer tokens at high feature quality
OCR depends heavily on resolution; reasoning depends on feature richness
Consider what happens during the LLM's attention: how does it use 2,304 vs. 256 tokens?

Papers

A Survey on Multimodal Large Language Models

Blogs

Which Vision Encoder for VLMs? — Hugging Face

CNN Feature Extractors

Vision Transformer (ViT)

CLIP Encoder

SigLIP

DINOv2

InternViT & Scaling

Choosing an Encoder

7 topics in this chapter

Convolutional Feature Extractors

In this topic

CNNs as Vision Backbones

Limitations of CNNs for VLMs

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank

Vision Transformer (ViT)

In this topic

Patch Embedding

Self-Attention on Image Patches

CLS Token and Feature Aggregation

Positional Encoding Strategies

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank

CLIP Visual Encoder

In this topic

Contrastive Pre-Training Objective

CLIP ViT-L/14: The Standard Backbone

Why CLIP Features Work for VLMs

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank

SigLIP

In this topic

Sigmoid Contrastive Loss

SigLIP-SO400M Architecture

SigLIP vs. CLIP: When to Choose Which

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank

InternViT & Scaling Vision Encoders

In this topic

InternViT-6B Architecture

Dynamic Resolution and Aspect Ratio Handling

Scaling Laws for Vision Encoders

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank

DINOv2

In this topic

Self-Distillation Framework

DINOv2 Feature Properties

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank

Choosing a Vision Encoder

In this topic

Frozen vs. Fine-Tuned Vision Encoders

Resolution vs. Token Count Trade-Off

Feature Granularity Comparison

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank