Loading...
Explore the vision encoders that power modern VLMs: from convolutional feature extractors to Vision Transformers, CLIP, SigLIP, DINOv2, and scaled architectures like InternViT. Understand how each encodes images into the feature representations that language models reason over.
The vision encoder is the first component in any VLM pipeline. Its job is to transform raw pixel data---a dense 3D tensor of RGB values---into a sequence of high-dimensional feature vectors that capture the semantic content of the image. The quality of these visual features determines the upper bound of what the entire VLM can perceive and reason about.
The evolution of vision encoders mirrors the broader history of computer vision:
Convolutional Neural Networks (CNNs): ResNet, EfficientNet, and their variants dominated from 2015 to 2020. They produce hierarchical feature maps through learned convolutional filters, capturing increasingly abstract patterns at each layer.
Vision Transformers (ViT): Dosovitskiy et al. (2020) showed that the same transformer architecture used for language could process images by treating image patches as tokens. ViTs have largely replaced CNNs as the standard vision backbone.
Contrastively-trained encoders: CLIP (2021) and SigLIP (2023) train ViTs to produce features aligned with text embeddings, creating a shared vision-language representation space. These are the most common encoders in modern VLMs.
Self-supervised encoders: DINOv2 (2023) trains ViTs without any text supervision, producing rich visual features through self-distillation. These features excel at dense prediction tasks.
Scaled encoders: InternViT-6B (2024) pushes the parameter count to 6 billion, demonstrating that scaling vision encoders improves VLM performance, analogous to scaling laws for LLMs.
The key trade-offs in choosing a vision encoder are:
where is the number of patches for image resolution and patch size .
Click any topic to jump in
How convolutional networks extract hierarchical visual features — the starting point before transformers.
Splitting images into patches and processing them with self-attention — the architecture that replaced CNNs for VLMs.
Contrastive, sigmoid contrastive, and self-supervised
Contrastive pre-training aligns vision features with language — the most common VLM backbone.
Sigmoid loss removes the need for global batch normalization — simpler and more scalable than CLIP.
Self-distillation without labels produces rich spatial features — complementary to contrastive encoders.
Scaling vision encoders to 6B parameters with dynamic resolution — pushing the frontier of visual understanding.
Frozen vs fine-tuned, resolution vs tokens, feature granularity — how to pick the right encoder for your VLM.
Before Vision Transformers, convolutional neural networks (CNNs) were the standard vision backbone. Models like ResNet, EfficientNet, and Inception produced hierarchical feature representations through stacked convolutional layers. Understanding CNNs provides essential context for why the field moved to transformers---and why some VLM designs still incorporate convolutional components.
A CNN processes an image through a series of convolutional layers, each applying learnable filters that detect patterns at increasing levels of abstraction:
The output of a CNN backbone is a feature map: a 3D tensor of shape where is the number of feature channels, and are the spatially downsampled dimensions. For ResNet-50 processing a image, the final feature map is ---2048 channels at spatial resolution.
For VLMs, this feature map can be flattened into a sequence of tokens of dimension 2048, then projected into the LLM's embedding space.
The feature map dimension follows while spatial extent shrinks as . The total information budget per layer stays roughly constant: . This means no information is created or destroyed — it is merely re-encoded from spatial to channel dimensions.
A ResNet-50 backbone produces a feature map of shape (2048, 7, 7) for a 224x224 input image. How does this compare to a ViT-L/14 processing the same image?
While CNNs remain useful in some contexts, they have fundamental limitations as VLM encoders:
Limited receptive field: Each CNN feature only captures a local region. Global context requires very deep stacking of layers, making it indirect.
Fixed spatial resolution: The feature map resolution is determined by the architecture. Changing input resolution requires interpolation or architectural changes.
No built-in token sequence: LLMs expect a sequence of tokens. Converting CNN feature maps to tokens is an afterthought rather than a native design.
No text alignment: CNNs are typically pre-trained on ImageNet classification, which provides category-level understanding but not the rich semantic features learned through text-aligned training (CLIP).
Translation equivariance, not invariance: Convolutions detect patterns regardless of position, which is useful for classification but means spatial information is partially lost---problematic for grounding tasks.
The effective receptive field (ERF) of a CNN grows as with depth , not as the theoretical receptive field suggests. After 50 layers of convolutions the ERF covers roughly of the image, meaning deep CNN features are still largely local. Self-attention achieves a global receptive field ( attention matrix) in a single layer.
Why did Bottom-Up Top-Down Attention (Anderson et al., 2018)---one of the most successful early VLM approaches---use Faster R-CNN instead of a standard ResNet classifier?
Explain why Vision Transformers replaced CNNs as the dominant vision encoder in VLMs despite CNNs having lower computational cost and decades of optimization. Identify at least three architectural properties of ViTs that make them inherently better suited for the VLM pipeline.