Chapter 1: Introduction to Vision-Language Models

Week 1-2

Understand what Vision-Language Models are, how they bridge the gap between visual perception and language understanding, and why the multimodal paradigm represents a fundamental shift in AI. Trace the evolution from early VQA systems to modern unified architectures like GPT-4o and Gemini.

Topics: 6
Demo: 1
Videos: 3
Min read: 45

Chapter Overview

Vision-Language Models (VLMs) represent a new frontier in artificial intelligence: systems that can simultaneously understand images and generate language about them. Unlike traditional computer vision models that output class labels or bounding boxes, VLMs produce rich, contextual natural language descriptions, answer open-ended questions about visual content, and reason across both modalities.

The core insight behind VLMs is deceptively simple: if we connect a powerful vision encoder to a powerful language model, the resulting system can leverage the reasoning abilities of the LLM while grounding its outputs in visual reality. The conditional generation objective captures this:

$P(\text{text} \mid \text{image}, \text{prompt}) = \prod_{t=1}^{T} P(y_t \mid y_{<t}, \text{image}, \text{prompt})$

This factorization means the model generates text one token at a time, conditioned on everything it has seen so far---both the image features and any preceding text tokens.

The field has evolved rapidly. Early approaches (2015--2020) treated vision-language tasks as classification problems, selecting from a fixed set of answers. The CLIP revolution (2021) demonstrated that contrastive pre-training could align visual and textual representations in a shared embedding space. Flamingo (2022) showed how to inject visual features into frozen language models. LLaVA (2023) proved that a simple linear projection plus visual instruction tuning could match far more complex architectures. And GPT-4V/4o (2023--2024) brought multimodal reasoning to production scale.

This chapter covers:

What are VLMs? The multimodal paradigm and conditional generation framework
The Vision-Language Gap: Why bridging modalities is fundamentally challenging
Evolution of VLMs: From VQA classifiers to native multimodal models
Architecture Overview: The three-component design pattern
Key Capabilities: What modern VLMs can actually do
Landscape of Modern VLMs: Closed vs. open models and how they compare

Chapter Roadmap

Click any topic to jump in

Explainer videoInteractive demo

What Are VLMs

The multimodal paradigm — why combining vision and language produces capabilities neither modality has alone.

The Multimodal ParadigmUnimodal vs. Multimodal ModelsConditional Generation Framework

The problem and the solution

Understanding the challenge and how architectures address it

The Vision-Language Gap

Why bridging pixels and words is fundamentally hard — grounding, modality mismatch, and alignment challenges.

The Grounding ProblemModality MismatchRepresentation Alignment

Architecture Overview

The three-component blueprint: vision encoder, bridge module, and language decoder working together.

Component 1: Vision EncoderComponent 2: Bridge Module (Connector)Component 3: Language Model DecoderEnd-to-End Data Flow

How the field evolved

Evolution of VLMs

From VQA classifiers to native multimodal models — four eras of increasingly capable systems.

Era 1: VQA Classifiers (2015--2019)Era 2: CLIP and Contrastive Pre-Training (2021)Era 3: Generative VLMs (2022--2023)Era 4: Native Multimodal Models (2024+)

What VLMs enable today

Capabilities and the models that deliver them

Key Capabilities

VQA, captioning, OCR, and visual grounding — what modern VLMs can actually do.

Visual Question Answering (VQA)Image Captioning and Dense DescriptionOptical Character Recognition (OCR) and Document UnderstandingVisual Grounding and Referring Expression Comprehension

Modern VLM Landscape

Closed vs open-weight models and how to choose — GPT-4o, Gemini, LLaVA, Qwen-VL and more.

Closed-Source VLMsOpen-Weight VLMsModel Selection Criteria

Topics

6 topics in this chapter

An image classifier can tell you that a photograph contains a cat; a language model can write fluent prose about cats it has never seen. Neither can answer "what is odd about this particular picture?", because the classifier's entire vocabulary is its label set and the language model has no access to the pixels. Closing that gap needs a model whose text generation is conditioned on visual input, which is exactly the objective the chapter introduction wrote as $P(\text{text} \mid \text{image}, \text{prompt})$ — keep that factorization at hand, because everything below unpacks it. We begin with the multimodal paradigm, making precise what it means for visual features $\mathbf{v}$ to occupy positions in the same context window as text tokens. We then contrast unimodal and multimodal systems, since where you fuse the two streams, at the end or at every layer, sets a hard ceiling on the cross-modal reasoning available. Finally we turn the idea into a trainable loss. This ordering matters: The Vision-Language Gap, the topic that follows, is a catalogue of everything this objective quietly assumes, and it only reads as difficult once you know what is being asked.

In this topic

1The Multimodal Paradigm
2Unimodal vs. Multimodal Models
3Conditional Generation Framework

1 of 3

The Multimodal Paradigm

$P(y_1, \ldots, y_T \mid \mathbf{v}, x_1, \ldots, x_S) = \prod_{t=1}^{T} P(y_t \mid y_{<t}, \mathbf{v}, \mathbf{x})$

A VLM performs conditional text generation where the conditioning context includes both visual features $\mathbf{v}$ (extracted from the image by a vision encoder) and textual tokens $\mathbf{x}$ (the user's prompt or question). The model autoregressively generates output tokens $y_1, \ldots, y_T$ , with each token conditioned on all previous tokens plus the full visual and textual context.

This is a strict generalization of a standard language model, which conditions only on text: $P(y_t \mid y_{<t}, \mathbf{x})$ . The visual features $\mathbf{v}$ act as additional context tokens that the language model attends to during generation.

Critically, the visual features are not raw pixels---they are high-dimensional embeddings produced by a vision encoder (typically a Vision Transformer). These embeddings capture semantic content at multiple levels of abstraction, from low-level textures to high-level object relationships.

Mathematical Intuition

The joint distribution $P(y_t \mid y_{<t}, \mathbf{v}, \mathbf{x})$ factorizes over a context window of size $N_v + S + t - 1$ , where $N_v$ is the number of visual tokens and $S$ is the prompt length. Each attention layer computes $\text{softmax}(QK^\top / \sqrt{d})V$ over this full context, meaning visual tokens participate in $O((N_v + S + T)^2)$ pairwise interactions. The critical insight: visual tokens are not a separate input channel — they occupy the same sequence positions as text tokens, making cross-modal attention an emergent property of standard self-attention rather than a separate mechanism.

Example:

A VLM receives an image of a stop sign partially covered by snow and the prompt "What does this sign say?" How does the model process this?

2 of 3

Unimodal vs. Multimodal Models

Traditional AI models are unimodal: they process a single data type. A ResNet classifies images. GPT generates text. A speech model transcribes audio. Each operates in its own modality silo.

Multimodal models break these silos by processing multiple data types jointly. VLMs specifically bridge vision and language, but the principle extends to audio, video, and other modalities. The key challenge is creating a shared representation space where information from different modalities can interact meaningfully.

There are two fundamental approaches:

Late fusion: Process each modality independently, then combine the final representations. Simple but loses fine-grained cross-modal interactions.
Early/mid fusion: Interleave information from both modalities during processing. More complex but enables richer cross-modal reasoning.

Mathematical Intuition

Late fusion computes $f(\mathbf{v}) \oplus g(\mathbf{x})$ where $\oplus$ is some combination operator applied to final representations. Early fusion computes $h(\mathbf{v}, \mathbf{x})$ where cross-modal interactions occur at every layer. The information-theoretic difference is that late fusion can capture at most $\min(I(\mathbf{v}; Z_v), I(\mathbf{x}; Z_x))$ bits of cross-modal mutual information (bounded by each encoder's bottleneck), while early fusion can capture up to $I(\mathbf{v}, \mathbf{x}; Z)$ — the full joint mutual information, preserving correlations that neither modality captures alone.

Example:

A doctor examines a chest X-ray and writes a radiology report. Is this task unimodal or multimodal? How does it map to a VLM?

3 of 3

Conditional Generation Framework

$\mathcal{L} = -\sum_{t=1}^{T} \log P_{\theta}(y_t \mid y_{<t}, \mathbf{v}, \mathbf{x})$

VLMs are trained by maximizing the conditional log-likelihood of the target text given the image and prompt. The training loss $\mathcal{L}$ is the negative log-probability of each ground-truth token, summed over the entire sequence.

During training, the model sees (image, prompt, response) triples. The image is encoded into visual tokens $\mathbf{v}$ , the prompt provides the textual context $\mathbf{x}$ , and the model learns to predict each token of the response. This is identical to standard language model training, except the context is augmented with visual features.

The gradient flows through the language model and (optionally) through the bridge module and vision encoder, depending on which components are frozen vs. trainable.

Mathematical Intuition

The training loss $\mathcal{L} = -\sum_t \log P_\theta(y_t \mid y_{<t}, \mathbf{v}, \mathbf{x})$ is a maximum likelihood estimator over the conditional distribution. By the chain rule, minimizing this is equivalent to minimizing $D_{\text{KL}}(P_{\text{data}} \| P_\theta)$ — the KL divergence between the true data distribution and the model. The gradient $\nabla_\theta \mathcal{L}$ flows through the LLM to the bridge module and optionally to the vision encoder, with the bridge receiving gradients proportional to $\partial \mathcal{L} / \partial \mathbf{h}_v$ — how much better predictions would be if the visual token representations changed.

Example:

During training, a VLM sees an image of a golden retriever playing in a park with the caption "A golden retriever chasing a frisbee on green grass." What does the loss computation look like?

Theory Exercise

Problem:

Explain why a VLM that processes image and text in a shared transformer (early fusion) might produce more nuanced descriptions than one that processes them separately and concatenates the results (late fusion). What types of visual reasoning tasks would most benefit from early fusion?

Hints:

Think about what information is available at each transformer layer with early vs. late fusion
Consider tasks that require fine-grained spatial reasoning, like 'What color is the object to the left of the cat?'
Think about how cross-attention between visual and textual tokens enables grounding

Papers

A Survey on Multimodal Large Language Models

Blogs

Introduction to Vision-Language Models — Hugging Face

In this topic

1The Grounding Problem
2Modality Mismatch
3Representation Alignment

1 of 3

The Grounding Problem

Grounding refers to connecting language symbols to their referents in the visual world. When a human reads "the red car on the left," they effortlessly locate the corresponding object in a scene. For a model, this requires learning a mapping from linguistic descriptions to spatial regions in an image.

The grounding problem has multiple levels of difficulty:

Object-level grounding: Mapping nouns to objects ("dog" $\rightarrow$ bounding box around the dog)
Attribute grounding: Mapping adjectives to properties ("red" $\rightarrow$ the color of a specific object)
Relation grounding: Mapping spatial/semantic relations ("on top of" $\rightarrow$ relative positions of two objects)
Action grounding: Mapping verbs to visual events ("running" $\rightarrow$ motion patterns)

Without effective grounding, a VLM may generate fluent but incorrect descriptions---a phenomenon known as hallucination.

Mathematical Intuition

Grounding requires learning a mapping $g: \mathcal{V} \times \mathcal{L} \to \mathcal{R}$ from (visual features, language description) to spatial regions. For object-level grounding, the model must solve an assignment problem: given $n$ detected regions $\{r_i\}$ and $m$ noun phrases $\{\ell_j\}$ , find the optimal matching $\pi^* = \arg\max_\pi \sum_j \text{sim}(r_{\pi(j)}, \ell_j)$ . Counting failures occur when the vision encoder's feature map has insufficient spatial resolution to separate nearby instances — two objects within the same $p \times p$ patch become a single token, losing the distinction.

Example:

A VLM is asked "How many cats are in this image?" for a photo containing 3 cats. It answers "2 cats." What type of grounding failure is this?

2 of 3

Modality Mismatch

$\mathbf{v} \in \mathbb{R}^{N \times d_v}, \quad \mathbf{x} \in \mathbb{R}^{S \times d_t}, \quad d_v \neq d_t$

Vision encoders and language models operate in different embedding spaces with different dimensionalities, different statistical properties, and different learned structures. The vision encoder's embedding space $\mathbb{R}^{d_v}$ is optimized for visual similarity (similar-looking objects are nearby), while the language model's space $\mathbb{R}^{d_t}$ is optimized for linguistic coherence (semantically related words are nearby).

Beyond dimensionality, there are deeper structural differences:

Information density: A single image token from a ViT-L/14 encodes a $14 \times 14$ pixel region with all its color, texture, and edge information. A single text token encodes one subword. Visual tokens are vastly more informationally dense.
Ordering: Text tokens have a strict sequential order. Image tokens have 2D spatial relationships. The bridge module must reconcile these.
Granularity: Text can describe a scene at any level of abstraction ("a room" vs. "a Victorian-era parlor with mahogany furniture"). Image features represent all details simultaneously.

Mathematical Intuition

Vision tokens live in $\mathbb{R}^{N \times d_v}$ while text tokens live in $\mathbb{R}^{S \times d_t}$ , with $d_v \neq d_t$ and fundamentally different statistical structure. A single visual token from ViT-L/14 encodes a $14 \times 14 \times 3 = 588$ -dimensional pixel patch, compressed to $d_v = 1024$ — an information density of $\sim 0.6$ bytes per dimension. A text token represents one of $V \approx 32{,}000$ discrete symbols — approximately $\log_2 V \approx 15$ bits of information. The bridge module must reconcile these wildly different information densities while preserving the spatial structure that 2D vision tokens encode but 1D text tokens do not.

Example:

A ViT-L/14 vision encoder produces 257 tokens of dimension 1024. A Llama-2-7B language model expects tokens of dimension 4096. What are the options for bridging this gap?

3 of 3

Representation Alignment

$\text{sim}(\mathbf{v}_i, \mathbf{t}_j) = \frac{\mathbf{v}_i \cdot \mathbf{t}_j}{\|\mathbf{v}_i\| \, \|\mathbf{t}_j\|}$

For a VLM to work, visual and textual representations must be aligned: an image of a dog and the text "a photo of a dog" should produce similar feature vectors. This alignment can be achieved through:

Contrastive pre-training (CLIP approach): Train the vision and text encoders jointly so that matching image-text pairs have high cosine similarity while non-matching pairs have low similarity.
Generative alignment (LLaVA approach): Fine-tune the bridge module and/or LLM so that visual features can be "understood" by the language model through next-token prediction on image-caption pairs.
Hybrid alignment: Use a CLIP-pretrained vision encoder (already partially aligned to language) and then further align through generative fine-tuning.

The quality of alignment directly determines the VLM's ability to ground language in visual content.

Mathematical Intuition

Contrastive alignment optimizes the InfoNCE objective, which is a lower bound on mutual information: $I(\mathbf{v}; \mathbf{t}) \geq \log N - \mathcal{L}_{\text{InfoNCE}}$ , where $N$ is the batch size. CLIP's temperature parameter $\tau = 0.07$ controls the sharpness of the softmax: $P(j \mid i) = \frac{\exp(\text{sim}(v_i, t_j)/\tau)}{\sum_k \exp(\text{sim}(v_i, t_k)/\tau)}$ . Lower $\tau$ creates sharper distributions that demand precise alignment. At $\tau = 0.07$ , a cosine similarity difference of just $0.14$ between the correct and incorrect pairs translates to a probability ratio of $e^{0.14/0.07} = e^2 \approx 7.4\times$ .

Example:

After CLIP pre-training, the cosine similarity between an image of a cat and the text "a photo of a cat" is 0.85. The similarity between the same image and "a photo of a dog" is 0.15. What does this tell us about alignment quality?

Theory Exercise

Problem:

A VLM correctly identifies objects in an image ("There is a dog, a ball, and a fence") but generates the incorrect spatial relationship ("The dog is behind the fence" when the dog is actually in front of the fence). Analyze which aspect of the vision-language gap is responsible and propose a training strategy to fix it.

Hints:

Consider which type of grounding (object, attribute, relation, action) is failing
Think about what information is lost when 2D spatial features are projected into a 1D token sequence
Consider how spatial relationship training data could be augmented

Papers

Grounding Language in Visual Perception

Blogs

The Grounding Problem in Multimodal AI — Lilian Weng

In this topic

1Era 1: VQA Classifiers (2015--2019)
2Era 2: CLIP and Contrastive Pre-Training (2021)
3Era 3: Generative VLMs (2022--2023)
4Era 4: Native Multimodal Models (2024+)

1 of 4

Era 1: VQA Classifiers (2015--2019)

The earliest vision-language models treated multimodal tasks as classification problems. For Visual Question Answering (VQA), models like VQA (Antol et al., 2015) and Bottom-Up Top-Down Attention (Anderson et al., 2018) extracted image features using a CNN (ResNet, Faster R-CNN), encoded the question with an LSTM, fused the two representations, and classified the answer from a fixed vocabulary of ~3,000 possible answers.

Limitations: Fixed answer vocabulary, no free-form generation, task-specific architectures. A model trained for VQA couldn't do captioning or visual reasoning without retraining.

Mathematical Intuition

VQA classifiers model $P(a \mid \mathbf{v}, \mathbf{q}) = \text{softmax}(W \cdot (f(\mathbf{v}) \odot g(\mathbf{q})))$ where $a \in \{1, \ldots, K\}$ indexes a fixed answer vocabulary of size $K \approx 3{,}000$ . The Hadamard product $\odot$ for fusion is a rank-1 bilinear interaction — it captures element-wise correlations but misses higher-order interactions between visual and textual features. The entropy of the output distribution is bounded by $\log K \approx 11.5$ bits, fundamentally limiting the expressiveness compared to generative models where output entropy scales with sequence length.

Example:

A VQA classifier is trained on the VQA v2 dataset with 3,129 answer classes. A user asks "What is the chemical formula written on the whiteboard?" How does this expose the model's limitations?

2 of 4

Era 2: CLIP and Contrastive Pre-Training (2021)

$\mathcal{L}_{\text{CLIP}} = -\frac{1}{2N} \sum_{i=1}^{N} \left[ \log \frac{e^{\text{sim}(v_i, t_i)/\tau}}{\sum_{j=1}^{N} e^{\text{sim}(v_i, t_j)/\tau}} + \log \frac{e^{\text{sim}(t_i, v_i)/\tau}}{\sum_{j=1}^{N} e^{\text{sim}(t_i, v_j)/\tau}} \right]$

CLIP (Contrastive Language-Image Pre-training, Radford et al., 2021) was a watershed moment. Instead of training on labeled datasets, CLIP trained on 400 million image-text pairs scraped from the internet. A vision encoder (ViT or ResNet) and a text encoder (Transformer) were trained jointly so that matching pairs had high cosine similarity.

CLIP demonstrated remarkable zero-shot transfer: without any task-specific fine-tuning, it could classify images, retrieve images from text queries, and perform visual reasoning. CLIP's vision encoder became the standard backbone for subsequent VLMs.

Key innovation: Learning visual representations from natural language supervision at web scale, rather than from manually labeled categories.

Mathematical Intuition

CLIP's symmetric InfoNCE loss over a batch of $N$ pairs creates $2N$ classification problems, each with $N$ classes. With batch size $N = 32{,}768$ , the effective learning signal is $2 \times 32{,}768 \times \log(32{,}768) \approx 10^6$ bits per batch. The learnable temperature $\tau$ (initialized at $0.07$ ) acts as an inverse bandwidth in the similarity kernel: $\kappa(v, t) = \exp(\cos(v, t) / \tau)$ . Training on 400M pairs with $\tau \to 0.01$ creates an embedding space where the cosine similarity margin between matching and non-matching pairs exceeds $0.5$ — enabling zero-shot transfer to downstream tasks without fine-tuning.

Example:

CLIP is given an image of a dog and the text candidates ["a photo of a cat", "a photo of a dog", "a photo of a car"]. How does zero-shot classification work?

3 of 4

Era 3: Generative VLMs (2022--2023)

The next leap came from connecting vision encoders to generative language models. Instead of just matching images and text, these models generate free-form text conditioned on images.

Flamingo (DeepMind, 2022): Inserted gated cross-attention layers into a frozen Chinchilla LLM, allowing it to attend to visual features from a frozen CLIP encoder. Demonstrated few-shot visual learning.
BLIP-2 (Salesforce, 2023): Introduced Q-Former, a lightweight transformer that bridges frozen vision and language models using learnable query tokens.
LLaVA (Liu et al., 2023): Showed that a simple linear projection from CLIP features to LLM token space, combined with visual instruction tuning, achieves competitive performance (LLaVA-1.5 later swapped it for a 2-layer MLP, still tiny next to the LLM). Dramatically simpler than alternatives.

Key insight: You don't need to train vision and language from scratch together. Connecting pre-trained frozen components with a small trainable bridge is sufficient.

Mathematical Intuition

The original LLaVA's bridge is a single linear projection $W \in \mathbb{R}^{d_v \times d_{\text{LLM}}}$ with only $d_v \times d_{\text{LLM}}$ parameters (e.g., $1024 \times 4096 \approx 4M$ for a 7B LLM). LLaVA-1.5's 2-layer MLP is still only tens of millions. This works because CLIP's pre-training already produces features where $\cos(\mathbf{v}, \mathbf{t}) > 0.8$ for matching pairs — the representations are approximately aligned up to a linear transform. Formally, if the CLIP feature spaces satisfy $\mathbf{v} \approx A\mathbf{t} + \epsilon$ for some unknown $A$ , then learning $W$ is a linear regression problem solvable with $\sim d_v \times d_{\text{LLM}}$ examples, far fewer than learning visual representations from scratch.

Example:

LLaVA-1.5 uses a 2-layer MLP with ~31M parameters (1024×5120 + 5120×5120) to connect CLIP ViT-L (304M params) to Vicuna-13B (13B params). How is it possible that such a small bridge works?

4 of 4

Era 4: Native Multimodal Models (2024+)

The most recent generation moves beyond the modular approach (frozen encoder + bridge + frozen LLM) toward models that are designed multimodal from the ground up:

GPT-4V/4o (OpenAI): Natively multimodal, trained end-to-end on text, images, and audio. Architecture details are proprietary.
Gemini (Google): Pre-trained from scratch on interleaved multimodal data---text, images, audio, video, code.
Claude 3/3.5 (Anthropic): Vision capabilities integrated into the language model architecture.

Key distinction: Earlier VLMs bolt vision onto a language model. Native multimodal models treat all modalities as first-class citizens during pre-training, potentially learning deeper cross-modal representations.

Mathematical Intuition

Native multimodal models optimize $P(\mathbf{y} \mid \mathbf{m}_1, \ldots, \mathbf{m}_K)$ jointly over $K$ modalities during pre-training, rather than freezing individual encoders and learning only the bridge. The key advantage is captured by the data processing inequality: for any Markov chain $\mathbf{v} \to f(\mathbf{v}) \to g(f(\mathbf{v}))$ , we have $I(\mathbf{v}; g(f(\mathbf{v}))) \leq I(\mathbf{v}; f(\mathbf{v}))$ . Modular approaches with frozen encoders form such a chain, meaning the bridge can never recover information discarded by the vision encoder. End-to-end training removes this bottleneck, allowing the vision encoder to preserve information that the LLM specifically needs.

Example:

What are the trade-offs between modular VLMs (LLaVA-style) and native multimodal models (Gemini-style)?

Theory Exercise

Problem:

Compare the training efficiency (compute required) and capability range of three approaches: (1) training a task-specific VQA classifier from scratch, (2) fine-tuning a CLIP model for VQA, and (3) visual instruction tuning an LLM (LLaVA approach). Under what circumstances would each be the best choice?

Hints:

Consider dataset size requirements for each approach
Think about what capabilities each approach provides beyond VQA
Consider computational budget constraints

Papers

VQA: Visual Question Answering Learning Transferable Visual Models From Natural Language Supervision (CLIP)Visual Instruction Tuning (LLaVA)

Blogs

A Decade of Vision-Language Models — Hugging Face

In this topic

1Component 1: Vision Encoder
2Component 2: Bridge Module (Connector)
3Component 3: Language Model Decoder
4End-to-End Data Flow

1 of 4

Component 1: Vision Encoder

The vision encoder transforms raw pixel input into a sequence of high-dimensional feature vectors. Modern VLMs almost universally use Vision Transformers (ViT) as their vision encoder, typically pre-trained with CLIP, SigLIP, or DINOv2.

The encoder divides the image into patches (e.g., $16 \times 16$ pixels), embeds each patch into a vector, adds positional information, and processes the sequence through transformer layers. The output is a set of visual tokens---one per patch plus optionally a CLS token---each capturing semantic information about its corresponding image region.

Common choices:

CLIP ViT-L/14: 304M params, 1024-dim, 256 tokens for 224px input. The standard choice.
SigLIP-SO400M: 400M params, efficient sigmoid-based contrastive training.
InternViT-6B: 6B params, the largest open vision encoder.
DINOv2 ViT-g: 1.1B params, self-supervised (no text alignment).

Mathematical Intuition

A ViT with patch size $p$ on an image of size $H \times W$ produces $N = \frac{H}{p} \times \frac{W}{p}$ tokens. Each patch is linearly projected: $\mathbf{z}_i = W_E \cdot \text{flatten}(\text{patch}_i) + \mathbf{e}_i^{\text{pos}}$ , where $W_E \in \mathbb{R}^{d \times p^2 c}$ . After $L$ transformer layers, the receptive field of each token grows to encompass the entire image through self-attention, but the spatial correspondence is preserved — token $i$ still primarily represents patch $i$ with contextual enrichment from all other patches. This is why positional embeddings matter: without them, the model sees an unordered set of patches and loses all spatial structure.

Example:

A ViT-L/14 processes a 336x336 image. How many visual tokens does it produce?

2 of 4

Component 2: Bridge Module (Connector)

$\mathbf{h}_\text{visual} = f_\text{bridge}(\mathbf{v}_1, \ldots, \mathbf{v}_N) \in \mathbb{R}^{M \times d_\text{LLM}}$

The bridge module maps visual tokens from the encoder's space ( $d_v$ ) to the LLM's embedding space ( $d_\text{LLM}$ ) and optionally compresses the number of tokens from $N$ to $M$ (where $M \leq N$ ). This is the most architecturally diverse component across different VLMs.

Common bridge architectures:

Linear Projection (LLaVA-v1): Single matrix $W \in \mathbb{R}^{d_v \times d_\text{LLM}}$ . Preserves all $N$ tokens. Simplest possible approach.
MLP Projection (LLaVA-v1.5): Two-layer MLP with GELU activation. Still preserves all $N$ tokens but with a more expressive transformation.
Perceiver Resampler (Flamingo): Cross-attention from $M$ learnable query tokens to $N$ visual tokens. Compresses from $N$ to $M$ tokens (e.g., 256 $\rightarrow$ 64).
Q-Former (BLIP-2): Lightweight transformer with learnable queries that extract the most relevant information from visual tokens through cross-attention.
C-Abstractor (Honeybee): Convolutional abstractor that uses convolutions to reduce spatial resolution before projection.

Mathematical Intuition

The Perceiver Resampler compresses $N$ visual tokens to $M$ query tokens ( $M \ll N$ ) via cross-attention: $\text{Output} = \text{softmax}(Q_\text{learnable} K_\text{visual}^\top / \sqrt{d}) V_\text{visual}$ , where $Q \in \mathbb{R}^{M \times d}$ and $K, V \in \mathbb{R}^{N \times d}$ . This is a learned bottleneck: the $M$ queries must discover which of the $N$ visual features are most informative for language generation. Information-theoretically, the bottleneck forces $I(\text{Output}; \text{Visual}) \leq M \cdot d \cdot \log(\text{precision})$ bits, trading visual detail for computational efficiency in the downstream LLM.

Example:

Compare the parameter counts and output token counts for a linear projection vs. a Perceiver Resampler when mapping 576 ViT-L tokens (dim=1024) to Llama-2-7B (dim=4096).

3 of 4

Component 3: Language Model Decoder

The language model is the core reasoning engine. It receives a sequence of tokens---visual tokens from the bridge module interleaved with text tokens from the user's prompt---and generates a response autoregressively.

Modern VLMs use decoder-only transformer LLMs as their backbone:

Vicuna/Llama: 7B--70B params. Most common in open-source VLMs (LLaVA, InternVL).
Qwen: 1.8B--72B params. Used in Qwen-VL series.
Phi-3: 3.8B params. Used in Phi-3-Vision for efficient deployment.
Mixtral: 8x7B MoE. Used for efficient scaling.

A critical design decision is whether the LLM is frozen (weights fixed from pre-training) or fine-tuned during VLM training. Freezing preserves the LLM's language capabilities but limits adaptation. Full fine-tuning allows deeper integration but risks catastrophic forgetting of language skills.

Mathematical Intuition

The LLM applies causal self-attention over the concatenated sequence $[\mathbf{h}_v; \mathbf{h}_t]$ , computing $\text{Attn}(Q, K, V) = \text{softmax}(QK^\top / \sqrt{d} + M_{\text{causal}})V$ where $M_{\text{causal}}$ masks future tokens. Visual tokens are never masked from each other (they are all "past" relative to text tokens), so the LLM can freely attend across all visual patches when processing the prompt. The decision to freeze vs. fine-tune the LLM affects gradient flow: frozen LLMs have $\nabla_{\theta_{\text{LLM}}} \mathcal{L} = 0$ , forcing all adaptation into the bridge module's $\sim 4$ - $300$ M parameters.

Example:

LLaVA-v1 freezes the LLM during pre-training and fine-tunes it during instruction tuning. LLaVA-v1.5 fine-tunes the LLM in both stages. Why does this matter?

4 of 4

End-to-End Data Flow

$\text{Output} = \text{LLM}([f_\text{bridge}(\text{ViT}(\text{image})); \text{Embed}(\text{prompt})])$

The complete forward pass concatenates visual and textual tokens into a single sequence that the LLM processes with its standard causal attention mechanism:

Image encoding: $\mathbf{v} = \text{ViT}(\text{image}) \in \mathbb{R}^{N \times d_v}$
Bridge projection: $\mathbf{h}_v = f_\text{bridge}(\mathbf{v}) \in \mathbb{R}^{M \times d_\text{LLM}}$
Text embedding: $\mathbf{h}_t = \text{Embed}(\text{prompt}) \in \mathbb{R}^{S \times d_\text{LLM}}$
Concatenation: $\mathbf{h} = [\mathbf{h}_v; \mathbf{h}_t] \in \mathbb{R}^{(M+S) \times d_\text{LLM}}$
Autoregressive generation: $y_t = \arg\max P(y_t \mid \mathbf{h}, y_{<t})$

From the LLM's perspective, visual tokens are indistinguishable from text tokens. They are just additional context in the input sequence. The LLM's self-attention mechanism naturally learns to attend to relevant visual tokens when generating each output word.

Mathematical Intuition

The total sequence length $L = M + S + T$ (visual + prompt + generated tokens) determines the computational cost: self-attention is $O(L^2 d)$ and the KV-cache grows as $O(L \cdot d \cdot n_{\text{layers}})$ . For a typical setup with $M = 576$ visual tokens, $S = 20$ prompt tokens, and $T = 200$ generated tokens, visual tokens constitute $576/796 \approx 72\%$ of the sequence but carry all the perceptual information. This motivates token compression: reducing $M$ from 576 to 64 cuts the prefill attention cost by $(576+20)^2 / (64+20)^2 \approx 50\times$ .

Example:

For a VLM with 576 visual tokens, a 20-token prompt, and a 100-token response, what is the total sequence length? How does this affect computational cost?

Theory Exercise

Problem:

Design a VLM for a mobile phone that must process high-resolution images (1344x1344) in under 2 seconds. Specify your choices for vision encoder, bridge module, and LLM, and justify each choice. What are the key trade-offs you must navigate?

Hints:

Consider the token count explosion with high-resolution inputs
Think about which component dominates latency: vision encoder, bridge, or LLM
Explore MoE (Mixture of Experts) LLMs for efficiency
Consider dynamic resolution strategies

Papers

Flamingo: a Visual Language Model for Few-Shot Learning

Blogs

The Architecture of Multimodal Models — Chip Huyen

In this topic

1Visual Question Answering (VQA)
2Image Captioning and Dense Description
3Optical Character Recognition (OCR) and Document Understanding
4Visual Grounding and Referring Expression Comprehension

1 of 4

Visual Question Answering (VQA)

The most fundamental VLM capability: given an image and a question, generate a natural language answer. Modern VLMs handle open-ended questions far beyond the fixed-vocabulary classifiers of earlier approaches.

Difficulty spectrum:

Recognition: "What animal is this?" (easy---single object identification)
Counting: "How many people are in the photo?" (medium---requires instance segmentation)
Spatial reasoning: "What is to the left of the red car?" (hard---requires precise spatial grounding)
Complex reasoning: "Based on the graph, what year had the highest GDP growth and what policy might explain this?" (very hard---requires reading, reasoning, and external knowledge)

Mathematical Intuition

VQA difficulty correlates with the number of reasoning hops required. Recognition is a single lookup: $a = \arg\max_c P(c \mid \mathbf{v})$ . Counting requires enumeration: $n = |\{i : P(\text{object}_i \in \text{class} \mid \mathbf{v}) > \theta\}|$ , which demands the model maintain distinct representations for each instance. Spatial reasoning requires composing relations: answering "What is left of the red car?" is $a = \arg\max_o P(o \mid \text{left\_of}(o, \text{select}(\text{red}, \text{car}, \mathbf{v})))$ — a two-hop query where errors compound multiplicatively across steps.

Example:

Rank these VQA questions by difficulty and explain why: (a) "Is there a dog in the image?" (b) "What breed of dog is in the image?" (c) "How does the dog's body language suggest it is feeling?"

2 of 4

Image Captioning and Dense Description

Generating textual descriptions of images, ranging from brief captions to detailed paragraphs. This was one of the earliest multimodal tasks, but modern VLMs have dramatically expanded what is possible.

Brief captioning: "A dog playing in a park" --- captures the main subject and action.
Detailed captioning: Multi-sentence descriptions covering objects, attributes, spatial relationships, background elements, lighting, and mood.
Region-specific captioning: Describing specific parts of an image ("What is in the top-right corner?").
Structured captioning: Generating JSON-formatted descriptions with attributes, relationships, and bounding boxes.

Mathematical Intuition

Caption generation follows $P(w_1, \ldots, w_T \mid \mathbf{v}) = \prod_t P(w_t \mid w_{<t}, \mathbf{v})$ , where caption quality depends on the attention pattern over visual tokens at each decoding step. Brief captions attend to high-saliency patches (often $< 10\%$ of tokens receive significant attention weight). Dense descriptions require near-uniform coverage — the cumulative attention distribution $\sum_t \alpha_t$ should have high entropy across all $N$ visual tokens. Hallucination occurs when the LLM's language prior $P(w_t \mid w_{<t})$ overwhelms the visual signal $P(w_t \mid w_{<t}, \mathbf{v})$ , generating plausible but ungrounded continuations.

Example:

Why is detailed image captioning harder than brief captioning for a VLM? Isn't generating more text just... generating more tokens?

3 of 4

Optical Character Recognition (OCR) and Document Understanding

Modern VLMs can read and understand text within images, including handwritten text, tables, charts, mathematical equations, code screenshots, and complex document layouts. This emerged as a surprisingly strong capability in GPT-4V and Gemini.

OCR capabilities include:

Reading printed and handwritten text
Understanding table structures and extracting data
Interpreting charts and graphs numerically
Parsing mathematical equations from LaTeX renderings or handwriting
Understanding multi-column layouts, headers, footers

This capability is particularly valuable because it means VLMs can process documents, receipts, forms, and screenshots without needing a separate OCR pipeline.

Mathematical Intuition

OCR in VLMs requires the vision encoder to preserve character-level detail across patches. For a ViT with patch size $p = 14$ at resolution $224 \times 224$ , each patch covers $14 \times 14$ pixels — roughly 2-3 characters of standard font. At this resolution, small or dense text falls below the Nyquist limit for reliable character discrimination. This is why high-resolution VLMs ( $\geq 672$ px) dramatically outperform standard-resolution ones on OCR benchmarks: doubling resolution quadruples the number of patches, giving each character $\sim 4\times$ more pixel area per token.

Example:

A VLM is given a screenshot of a bar chart showing quarterly revenue for 4 companies. What must the model do to answer "Which company had the highest Q3 revenue?"

4 of 4

Visual Grounding and Referring Expression Comprehension

Visual grounding is the task of localizing specific regions in an image based on a textual description. Given a description like "the person wearing a red hat," the model must output coordinates (a bounding box or a set of points) indicating where that entity is in the image.

Related tasks:

Referring Expression Comprehension (REC): Text $\rightarrow$ bounding box
Referring Expression Generation (REG): Bounding box $\rightarrow$ text description
Phrase grounding: Multiple phrases in a sentence, each mapped to a region

This capability requires fine-grained alignment between language tokens and visual patches, making it one of the hardest VLM tasks. Models like Kosmos-2, Shikra, and Ferret are specifically designed for grounding.

Mathematical Intuition

Visual grounding maps a referring expression $\ell$ to a bounding box $b = (x_1, y_1, x_2, y_2) \in [0,1]^4$ in normalized image coordinates. The model must learn $P(b \mid \ell, \mathbf{v})$ , typically by predicting coordinate tokens discretized into $N_{\text{bins}}$ bins per axis. With $N_{\text{bins}} = 1000$ , the spatial precision is $\sim 0.1\%$ of the image dimension. The key challenge is that grounding requires attending to a specific subset of visual tokens corresponding to the referred region — the attention pattern must be spatially localized rather than distributed, which is the opposite of what general captioning requires.

Example:

A user says to a VLM: "Circle the tallest building in this city skyline photo." What architectural capability does the VLM need that a standard captioning model lacks?

Theory Exercise

Problem:

Design an evaluation benchmark for testing VLM capabilities across five dimensions: recognition, spatial reasoning, OCR, counting, and world knowledge. For each dimension, create one easy, one medium, and one hard example question (with a reference image description). Explain what makes each level progressively harder.

Hints:

Consider what additional information/skills are needed at each difficulty level
Think about edge cases that would distinguish truly capable VLMs from those that just guess common answers
Consider multi-step reasoning chains for hard questions

Papers

Visual Instruction Tuning (LLaVA)

Blogs

Vision Language Models Explained — Encord

In this topic

1Closed-Source VLMs
2Open-Weight VLMs
3Model Selection Criteria

1 of 3

Closed-Source VLMs

These models are available only through APIs, with architecture details undisclosed:

GPT-4o (OpenAI): Currently the strongest general-purpose VLM. Natively multimodal (text, images, audio, video). Excels at complex reasoning, OCR, and instruction following.
Gemini 1.5 Pro (Google): Long-context multimodal (up to 1M tokens). Strong at video understanding and document processing.
Claude 3.5 Sonnet (Anthropic): Strong reasoning, excellent at code generation from screenshots, and careful about hallucination.
Gemini 2.0 Flash (Google): Optimized for speed while maintaining strong multimodal capabilities.

Key advantage: State-of-the-art performance, continuously updated, no infrastructure management. Key limitation: No fine-tuning (except GPT-4o), no architecture access, cost per token, data privacy concerns.

Mathematical Intuition

Closed-source VLMs like GPT-4o are estimated to use Mixture-of-Experts (MoE) architectures with $\sim 1.8$ T total parameters but only $\sim 200$ B active per token. The MoE routing function $g(x) = \text{TopK}(\text{softmax}(W_g x), k)$ selects $k$ out of $E$ experts per token, giving computational cost $O(k/E)$ relative to a dense model of the same total size. This explains how these models achieve superior reasoning (large total parameter count = more stored knowledge) while maintaining acceptable inference latency (small active parameter count = manageable FLOPs per token).

Example:

A healthcare company wants to build a VLM that reads medical images (X-rays, MRIs) and generates reports. Should they use GPT-4o or an open model?

2 of 3

Open-Weight VLMs

Models with publicly available weights, enabling fine-tuning, deployment, and inspection:

LLaVA-NeXT / LLaVA-OneVision (Bytedance/UW): Pioneer of visual instruction tuning. Simple architecture, strong results, easy to fine-tune. 7B--72B variants.
Qwen-VL / Qwen2-VL (Alibaba): State-of-the-art open VLM. Dynamic resolution, supports video. 2B--72B variants.
InternVL 2.5 (Shanghai AI Lab): Largest open VLM (InternViT-6B + InternLM2). Excels at OCR, document understanding.
Molmo (AI2): Focus on grounding and pointing. Open training data and pipeline.
Pixtral (Mistral): Variable-resolution via native patch handling. 12B params.
Phi-3-Vision (Microsoft): Efficient 4.2B model suitable for edge deployment.
Llama 3.2 Vision (Meta): 11B and 90B variants with strong performance across benchmarks.

Key advantage: Full control, fine-tuning, on-premises deployment, transparency. Key limitation: Generally behind closed models on complex reasoning tasks.

Mathematical Intuition

Open VLMs achieve competitive performance despite $200\times$ fewer parameters through architectural efficiency. Qwen2-VL's dynamic resolution encodes an image of size $H \times W$ with exactly $\lceil H/14 \rceil \times \lceil W/14 \rceil$ tokens — no padding waste. Combined with the naive resolution-scaling law that VLM performance improves as $\sim \log(\text{resolution})$ , this means doubling resolution costs $4\times$ more tokens but yields only $\sim \log 2 \approx 0.69$ units of improvement. The practical optimum balances this diminishing return against the LLM's $O(L^2)$ attention cost, typically landing at $672$ - $1344$ px for 7B-class models.

Example:

Compare LLaVA-NeXT-34B, Qwen2-VL-7B, and InternVL2.5-8B for a document understanding task that requires reading tables from scanned PDFs. Which would you choose?

3 of 3

Model Selection Criteria

Choosing the right VLM depends on several factors:

Criterion	Closed (API)	Open (Self-hosted)
Performance	Typically best	Close to SOTA
Cost	Per-token pricing	Fixed GPU cost
Privacy	Data leaves network	Fully on-premises
Fine-tuning	Limited/none	Full control
Latency	Variable (network)	Predictable
Maintenance	None	Significant

Decision framework:

Prototyping / research: Use GPT-4o API for best quality
Production (non-sensitive data): Evaluate cost of API vs. self-hosting at your volume
Production (sensitive data): Open model, on-premises
Edge/mobile: Phi-3-Vision or quantized Qwen2-VL-2B
Domain-specific: Fine-tune open model on domain data

Mathematical Intuition

The cost-efficiency frontier follows an approximate power law: $\text{Score} \propto \text{Cost}^\alpha$ with $\alpha \approx 0.1$ - $0.15$ across current VLMs. This means a $10\times$ increase in per-query cost (e.g., switching from Qwen2-VL-7B to GPT-4o) yields only $\sim 10^{0.12} \approx 1.3\times$ improvement in benchmark score. For volume-sensitive applications processing $N$ queries per month, the break-even point where self-hosting becomes cheaper than API is approximately $N^* = C_{\text{GPU}} / (c_{\text{API}} - c_{\text{self}})$ , where $C_{\text{GPU}}$ is the monthly GPU cost and $c_{\text{API}}, c_{\text{self}}$ are per-query costs.

Example:

A startup wants to build a product that analyzes fashion photos and suggests outfit improvements. They have 10,000 fashion expert-annotated image-text pairs. What VLM strategy should they use?

Theory Exercise

Problem:

You are building a visual search engine for an e-commerce platform with 50 million product images. Users upload a photo and expect to find similar products. Evaluate three approaches: (1) CLIP-based image retrieval, (2) VLM-based description + text search, (3) Hybrid approach. Consider latency, accuracy, and infrastructure requirements.

Hints:

Think about the index size for 50M images with each approach
Consider whether real-time VLM inference is feasible at search-time
Think about how text search can handle visual similarity that is hard to describe

Papers

GPT-4 Technical Report Gemini: A Family of Highly Capable Multimodal Models

Blogs

Open VLM Leaderboard — Hugging Face

What Are VLMs

The Vision-Language Gap

Architecture Overview

Evolution of VLMs

Key Capabilities

Modern VLM Landscape

6 topics in this chapter

What Are Vision-Language Models

In this topic

The Multimodal Paradigm

Unimodal vs. Multimodal Models

Conditional Generation Framework

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank

The Vision-Language Gap

In this topic

The Grounding Problem

Modality Mismatch

Representation Alignment

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank

Evolution of VLMs

In this topic

Era 1: VQA Classifiers (2015--2019)

Era 2: CLIP and Contrastive Pre-Training (2021)

Era 3: Generative VLMs (2022--2023)

Era 4: Native Multimodal Models (2024+)

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank

VLM Architecture Overview

In this topic

Component 1: Vision Encoder

Component 2: Bridge Module (Connector)

Component 3: Language Model Decoder

End-to-End Data Flow

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank

Key Capabilities

In this topic

Visual Question Answering (VQA)

Image Captioning and Dense Description

Optical Character Recognition (OCR) and Document Understanding

Visual Grounding and Referring Expression Comprehension

Theory Exercise

Problem:

Hints:

Related Problems on PixelBank

Landscape of Modern VLMs

In this topic

Closed-Source VLMs

Open-Weight VLMs

Model Selection Criteria

Theory Exercise

Problem:

Hints: