PIXELBANKv8.2.1
Menu
Sign In
Back to VLM Study Plan
Week 1-2

Chapter 1: Introduction to Vision-Language Models

Understand what Vision-Language Models are, how they bridge the gap between visual perception and language understanding, and why the multimodal paradigm represents a fundamental shift in AI. Trace the evolution from early VQA systems to modern unified architectures like GPT-4o and Gemini.

Chapter Overview

Vision-Language Models (VLMs) represent a new frontier in artificial intelligence: systems that can simultaneously understand images and generate language about them. Unlike traditional computer vision models that output class labels or bounding boxes, VLMs produce rich, contextual natural language descriptions, answer open-ended questions about visual content, and reason across both modalities.

The core insight behind VLMs is deceptively simple: if we connect a powerful vision encoder to a powerful language model, the resulting system can leverage the reasoning abilities of the LLM while grounding its outputs in visual reality. The conditional generation objective captures this:

P(text∣image,prompt)=∏t=1TP(yt∣y<t,image,prompt)P(\text{text} \mid \text{image}, \text{prompt}) = \prod_{t=1}^{T} P(y_t \mid y_{<t}, \text{image}, \text{prompt})

This factorization means the model generates text one token at a time, conditioned on everything it has seen so far---both the image features and any preceding text tokens.

The field has evolved rapidly. Early approaches (2015--2020) treated vision-language tasks as classification problems, selecting from a fixed set of answers. The CLIP revolution (2021) demonstrated that contrastive pre-training could align visual and textual representations in a shared embedding space. Flamingo (2022) showed how to inject visual features into frozen language models. LLaVA (2023) proved that a simple linear projection plus visual instruction tuning could match far more complex architectures. And GPT-4V/4o (2023--2024) brought multimodal reasoning to production scale.

This chapter covers:

  • What are VLMs? The multimodal paradigm and conditional generation framework
  • The Vision-Language Gap: Why bridging modalities is fundamentally challenging
  • Evolution of VLMs: From VQA classifiers to native multimodal models
  • Architecture Overview: The three-component design pattern
  • Key Capabilities: What modern VLMs can actually do
  • Landscape of Modern VLMs: Closed vs. open models and how they compare

Chapter Roadmap

Click any topic to jump in

1
What Are VLMs

The multimodal paradigm — why combining vision and language produces capabilities neither modality has alone.

The Multimodal ParadigmUnimodal vs. Multimodal ModelsConditional Generation Framework
The problem and the solution

Understanding the challenge and how architectures address it

2
The Vision-Language Gap

Why bridging pixels and words is fundamentally hard — grounding, modality mismatch, and alignment challenges.

The Grounding ProblemModality MismatchRepresentation Alignment
3
Architecture Overview

The three-component blueprint: vision encoder, bridge module, and language decoder working together.

Component 1: Vision EncoderComponent 2: Bridge Module (Connector)Component 3: Language Model DecoderEnd-to-End Data Flow
How the field evolved
4
Evolution of VLMs

From VQA classifiers to native multimodal models — four eras of increasingly capable systems.

Era 1: VQA Classifiers (2015--2019)Era 2: CLIP and Contrastive Pre-Training (2021)Era 3: Generative VLMs (2022--2023)Era 4: Native Multimodal Models (2024+)
What VLMs enable today

Capabilities and the models that deliver them

5
Key Capabilities

VQA, captioning, OCR, and visual grounding — what modern VLMs can actually do.

Visual Question Answering (VQA)Image Captioning and Dense DescriptionOptical Character Recognition (OCR) and Document UnderstandingVisual Grounding and Referring Expression Comprehension
6
Modern VLM Landscape

Closed vs open-weight models and how to choose — GPT-4o, Gemini, LLaVA, Qwen-VL and more.

Closed-Source VLMsOpen-Weight VLMsModel Selection Criteria

An image classifier can tell you that a photograph contains a cat; a language model can write fluent prose about cats it has never seen. Neither can answer "what is odd about this particular picture?", because the classifier's entire vocabulary is its label set and the language model has no access to the pixels. Closing that gap needs a model whose text generation is conditioned on visual input, which is exactly the objective the chapter introduction wrote as P(text∣image,prompt)P(\text{text} \mid \text{image}, \text{prompt}) — keep that factorization at hand, because everything below unpacks it. We begin with the multimodal paradigm, making precise what it means for visual features v\mathbf{v} to occupy positions in the same context window as text tokens. We then contrast unimodal and multimodal systems, since where you fuse the two streams, at the end or at every layer, sets a hard ceiling on the cross-modal reasoning available. Finally we turn the idea into a trainable loss. This ordering matters: The Vision-Language Gap, the topic that follows, is a catalogue of everything this objective quietly assumes, and it only reads as difficult once you know what is being asked.

In this topic

1The Multimodal Paradigm
2Unimodal vs. Multimodal Models
3Conditional Generation Framework
1 of 3
The Multimodal Paradigm

P(y1,…,yT∣v,x1,…,xS)=∏t=1TP(yt∣y<t,v,x)P(y_1, \ldots, y_T \mid \mathbf{v}, x_1, \ldots, x_S) = \prod_{t=1}^{T} P(y_t \mid y_{<t}, \mathbf{v}, \mathbf{x})

A VLM performs conditional text generation where the conditioning context includes both visual features v\mathbf{v} (extracted from the image by a vision encoder) and textual tokens x\mathbf{x} (the user's prompt or question). The model autoregressively generates output tokens y1,…,yTy_1, \ldots, y_T, with each token conditioned on all previous tokens plus the full visual and textual context.

This is a strict generalization of a standard language model, which conditions only on text: P(yt∣y<t,x)P(y_t \mid y_{<t}, \mathbf{x}). The visual features v\mathbf{v} act as additional context tokens that the language model attends to during generation.

Critically, the visual features are not raw pixels---they are high-dimensional embeddings produced by a vision encoder (typically a Vision Transformer). These embeddings capture semantic content at multiple levels of abstraction, from low-level textures to high-level object relationships.

Mathematical Intuition

The joint distribution P(yt∣y<t,v,x)P(y_t \mid y_{<t}, \mathbf{v}, \mathbf{x}) factorizes over a context window of size Nv+S+t−1N_v + S + t - 1, where NvN_v is the number of visual tokens and SS is the prompt length. Each attention layer computes softmax(QK⊤/d)V\text{softmax}(QK^\top / \sqrt{d})V over this full context, meaning visual tokens participate in O((Nv+S+T)2)O((N_v + S + T)^2) pairwise interactions. The critical insight: visual tokens are not a separate input channel — they occupy the same sequence positions as text tokens, making cross-modal attention an emergent property of standard self-attention rather than a separate mechanism.

Example:

A VLM receives an image of a stop sign partially covered by snow and the prompt "What does this sign say?" How does the model process this?

2 of 3
Unimodal vs. Multimodal Models

Traditional AI models are unimodal: they process a single data type. A ResNet classifies images. GPT generates text. A speech model transcribes audio. Each operates in its own modality silo.

Multimodal models break these silos by processing multiple data types jointly. VLMs specifically bridge vision and language, but the principle extends to audio, video, and other modalities. The key challenge is creating a shared representation space where information from different modalities can interact meaningfully.

There are two fundamental approaches:

  • Late fusion: Process each modality independently, then combine the final representations. Simple but loses fine-grained cross-modal interactions.
  • Early/mid fusion: Interleave information from both modalities during processing. More complex but enables richer cross-modal reasoning.
Mathematical Intuition

Late fusion computes f(v)⊕g(x)f(\mathbf{v}) \oplus g(\mathbf{x}) where ⊕\oplus is some combination operator applied to final representations. Early fusion computes h(v,x)h(\mathbf{v}, \mathbf{x}) where cross-modal interactions occur at every layer. The information-theoretic difference is that late fusion can capture at most min⁡(I(v;Zv),I(x;Zx))\min(I(\mathbf{v}; Z_v), I(\mathbf{x}; Z_x)) bits of cross-modal mutual information (bounded by each encoder's bottleneck), while early fusion can capture up to I(v,x;Z)I(\mathbf{v}, \mathbf{x}; Z) — the full joint mutual information, preserving correlations that neither modality captures alone.

Example:

A doctor examines a chest X-ray and writes a radiology report. Is this task unimodal or multimodal? How does it map to a VLM?

3 of 3
Conditional Generation Framework

L=−∑t=1Tlog⁡Pθ(yt∣y<t,v,x)\mathcal{L} = -\sum_{t=1}^{T} \log P_{\theta}(y_t \mid y_{<t}, \mathbf{v}, \mathbf{x})

VLMs are trained by maximizing the conditional log-likelihood of the target text given the image and prompt. The training loss L\mathcal{L} is the negative log-probability of each ground-truth token, summed over the entire sequence.

During training, the model sees (image, prompt, response) triples. The image is encoded into visual tokens v\mathbf{v}, the prompt provides the textual context x\mathbf{x}, and the model learns to predict each token of the response. This is identical to standard language model training, except the context is augmented with visual features.

The gradient flows through the language model and (optionally) through the bridge module and vision encoder, depending on which components are frozen vs. trainable.

Mathematical Intuition

The training loss L=−∑tlog⁡Pθ(yt∣y<t,v,x)\mathcal{L} = -\sum_t \log P_\theta(y_t \mid y_{<t}, \mathbf{v}, \mathbf{x}) is a maximum likelihood estimator over the conditional distribution. By the chain rule, minimizing this is equivalent to minimizing DKL(Pdata∥Pθ)D_{\text{KL}}(P_{\text{data}} \| P_\theta) — the KL divergence between the true data distribution and the model. The gradient ∇θL\nabla_\theta \mathcal{L} flows through the LLM to the bridge module and optionally to the vision encoder, with the bridge receiving gradients proportional to ∂L/∂hv\partial \mathcal{L} / \partial \mathbf{h}_v — how much better predictions would be if the visual token representations changed.

Example:

During training, a VLM sees an image of a golden retriever playing in a park with the caption "A golden retriever chasing a frisbee on green grass." What does the loss computation look like?

Theory Exercise

Problem:

Explain why a VLM that processes image and text in a shared transformer (early fusion) might produce more nuanced descriptions than one that processes them separately and concatenates the results (late fusion). What types of visual reasoning tasks would most benefit from early fusion?

Hints:
  • Think about what information is available at each transformer layer with early vs. late fusion
  • Consider tasks that require fine-grained spatial reasoning, like 'What color is the object to the left of the cat?'
  • Think about how cross-attention between visual and textual tokens enables grounding
Papers
A Survey on Multimodal Large Language Models
Blogs
Introduction to Vision-Language Models — Hugging Face
Next
Vision Encoders