Week 3-4

Chapter 2: Vision Encoders

Explore the vision encoders that power modern VLMs: from convolutional feature extractors to Vision Transformers, CLIP, SigLIP, DINOv2, and scaled architectures like InternViT. Understand how each encodes images into the feature representations that language models reason over.

Chapter Overview

The vision encoder is the first component in any VLM pipeline. Its job is to transform raw pixel data---a dense 3D tensor of RGB values---into a sequence of high-dimensional feature vectors that capture the semantic content of the image. The quality of these visual features determines the upper bound of what the entire VLM can perceive and reason about.

The evolution of vision encoders mirrors the broader history of computer vision:

Convolutional Neural Networks (CNNs): ResNet, EfficientNet, and their variants dominated from 2015 to 2020. They produce hierarchical feature maps through learned convolutional filters, capturing increasingly abstract patterns at each layer.
Vision Transformers (ViT): Dosovitskiy et al. (2020) showed that the same transformer architecture used for language could process images by treating image patches as tokens. ViTs have largely replaced CNNs as the standard vision backbone.
Contrastively-trained encoders: CLIP (2021) and SigLIP (2023) train ViTs to produce features aligned with text embeddings, creating a shared vision-language representation space. These are the most common encoders in modern VLMs.
Self-supervised encoders: DINOv2 (2023) trains ViTs without any text supervision, producing rich visual features through self-distillation. These features excel at dense prediction tasks.
Scaled encoders: InternViT-6B (2024) pushes the parameter count to 6 billion, demonstrating that scaling vision encoders improves VLM performance, analogous to scaling laws for LLMs.

The key trade-offs in choosing a vision encoder are:

Feature quality: How much semantic information is captured per patch token?
Resolution handling: Can the encoder process high-resolution images without quadratic cost explosion?
Alignment to language: Are the features already in a space compatible with text representations?
Computational cost: How many FLOPs per image, and how many output tokens?

$\text{Visual Tokens} = \text{ViT}(\text{Patchify}(\mathbf{I})) \in \mathbb{R}^{N \times d}$

where $N = (H / p) \times (W / p)$ is the number of patches for image resolution $H \times W$ and patch size $p$ .

Chapter Roadmap

Click any topic to jump in

CNN Feature Extractors

How convolutional networks extract hierarchical visual features — the starting point before transformers.

CNNs as Vision BackbonesLimitations of CNNs for VLMs

From convolutions to attention

Vision Transformer (ViT)

Splitting images into patches and processing them with self-attention — the architecture that replaced CNNs for VLMs.

Patch EmbeddingSelf-Attention on Image PatchesCLS Token and Feature AggregationPositional Encoding Strategies

Three training paradigms for ViTs

Contrastive, sigmoid contrastive, and self-supervised

CLIP Encoder

Contrastive pre-training aligns vision features with language — the most common VLM backbone.

Contrastive Pre-Training ObjectiveCLIP ViT-L/14: The Standard BackboneWhy CLIP Features Work for VLMs

SigLIP

Sigmoid loss removes the need for global batch normalization — simpler and more scalable than CLIP.

Sigmoid Contrastive LossSigLIP-SO400M ArchitectureSigLIP vs. CLIP: When to Choose Which

DINOv2

Self-distillation without labels produces rich spatial features — complementary to contrastive encoders.

Self-Distillation FrameworkDINOv2 Feature Properties

Scaling to billions of parameters

InternViT & Scaling

Scaling vision encoders to 6B parameters with dynamic resolution — pushing the frontier of visual understanding.

InternViT-6B ArchitectureDynamic Resolution and Aspect Ratio HandlingScaling Laws for Vision Encoders

Practical decision framework

Choosing an Encoder

Frozen vs fine-tuned, resolution vs tokens, feature granularity — how to pick the right encoder for your VLM.

Frozen vs. Fine-Tuned Vision EncodersResolution vs. Token Count Trade-OffFeature Granularity Comparison

An LLM consumes a sequence of vectors, but a photograph arrives as a dense grid of RGB intensities with no notion of a token, and nothing in the language model knows what to do with it. The chapter introduction framed the vision encoder as the component that closes that gap, and for roughly five years the only serious answer was a convolutional network. Before you can judge the Vision Transformer that follows, you need a concrete picture of what it displaced: how stacked convolutions build a hierarchy from edges to textures to whole objects, what a feature map of shape $(C, H', W')$ actually contains, and how that tensor gets flattened into something an LLM can read. The order here is deliberate. First the mechanics of CNN backbones and their output shapes, then the specific limitations that pushed the field elsewhere: local receptive fields, fixed resolution, tokens as an afterthought, no text alignment, and spatial precision traded away. This is not archaeology. Region-based CNN features powered the first generation of VQA systems, and hybrid designs still borrow convolutional stems today.

In this topic

1CNNs as Vision Backbones

2Limitations of CNNs for VLMs

1 of 2

CNNs as Vision Backbones

A CNN processes an image through a series of convolutional layers, each applying learnable filters that detect patterns at increasing levels of abstraction:

Early layers: Detect edges, textures, and color gradients (local patterns)
Middle layers: Detect parts of objects (eyes, wheels, windows)
Deep layers: Detect entire objects and scenes (faces, cars, rooms)

The output of a CNN backbone is a feature map: a 3D tensor of shape $(C, H', W')$ where $C$ is the number of feature channels, and $H', W'$ are the spatially downsampled dimensions. For ResNet-50 processing a $224 \times 224$ image, the final feature map is $(2048, 7, 7)$ ---2048 channels at $7 \times 7$ spatial resolution.

For VLMs, this feature map can be flattened into a sequence of $7 \times 7 = 49$ tokens of dimension 2048, then projected into the LLM's embedding space.

Mathematical Intuition

The feature map dimension follows $C_{\ell} \propto 2^{\ell}$ while spatial extent shrinks as $H_{\ell} = H_0 / 2^{\ell}$ . The total information budget per layer stays roughly constant: $C_{\ell} \cdot H_{\ell}^2 \approx C_0 \cdot H_0^2$ . This means no information is created or destroyed — it is merely re-encoded from spatial to channel dimensions.

Example:

A ResNet-50 backbone produces a feature map of shape (2048, 7, 7) for a 224x224 input image. How does this compare to a ViT-L/14 processing the same image?

2 of 2

Limitations of CNNs for VLMs

While CNNs remain useful in some contexts, they have fundamental limitations as VLM encoders:

Limited receptive field: Each CNN feature only captures a local region. Global context requires very deep stacking of layers, making it indirect.
Fixed spatial resolution: The feature map resolution is determined by the architecture. Changing input resolution requires interpolation or architectural changes.
No built-in token sequence: LLMs expect a sequence of tokens. Converting CNN feature maps to tokens is an afterthought rather than a native design.
No text alignment: CNNs are typically pre-trained on ImageNet classification, which provides category-level understanding but not the rich semantic features learned through text-aligned training (CLIP).
Translation equivariance, not invariance: Convolutions detect patterns regardless of position, which is useful for classification but means spatial information is partially lost---problematic for grounding tasks.

Mathematical Intuition

The effective receptive field (ERF) of a CNN grows as $O(\sqrt{L})$ with depth $L$ , not $O(L)$ as the theoretical receptive field suggests. After 50 layers of $3 \times 3$ convolutions the ERF covers roughly $\frac{1}{4}$ of the image, meaning deep CNN features are still largely local. Self-attention achieves a global receptive field ( $N \times N$ attention matrix) in a single layer.

Example:

Why did Bottom-Up Top-Down Attention (Anderson et al., 2018)---one of the most successful early VLM approaches---use Faster R-CNN instead of a standard ResNet classifier?

Theory Exercise

Problem:

Explain why Vision Transformers replaced CNNs as the dominant vision encoder in VLMs despite CNNs having lower computational cost and decades of optimization. Identify at least three architectural properties of ViTs that make them inherently better suited for the VLM pipeline.

Hints:

Consider how self-attention creates global receptive fields vs. local convolutions
Think about how ViT outputs naturally map to the token-based interface of LLMs
Consider the alignment between ViT training objectives (CLIP) and VLM requirements

Papers

Deep Residual Learning for Image Recognition (ResNet)

Blogs

CNN Explainer — Interactive Visualization

Introduction to Vision-Language Models

Language Model Backbone

Chapter 2: Vision Encoders

Chapter Overview

Chapter Roadmap

CNN Feature Extractors

Vision Transformer (ViT)

CLIP Encoder

SigLIP

DINOv2

InternViT & Scaling

Choosing an Encoder

Convolutional Feature Extractors

In this topic

CNNs as Vision Backbones

Limitations of CNNs for VLMs

Theory Exercise

Problem:

Hints:

Vision Transformer (ViT)

CLIP Visual Encoder

SigLIP

InternViT & Scaling Vision Encoders

DINOv2

Choosing a Vision Encoder