Paper breakdownMultimodalFREE2023

LLaVA: Visual Instruction Tuning

Large Language-and-Vision Assistant

Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee

Concepts

Formulas

Demos

Paper Overview

LLaVA (Large Language-and-Vision Assistant) introduced a surprisingly simple yet powerful recipe for building multimodal AI: connect a pre-trained vision encoder (CLIP ViT-L/14) to a large language model (Vicuna/LLaMA) through a lightweight projection layer, then train in two stages — first align visual and language representations, then instruction-tune for general-purpose visual understanding.

The core insight: You don't need complex cross-attention modules, Q-Formers, or perceiver architectures to build a strong vision-language model. A simple linear projection (LLaVA 1.0) or 2-layer MLP (LLaVA 1.5) connecting frozen CLIP features to the LLM's embedding space is sufficient — when combined with high-quality instruction-following data.

A second key contribution: Using language-only GPT-4 to generate multimodal instruction-following data from image captions and bounding boxes. This produced 158K high-quality training samples (conversations, detailed descriptions, complex reasoning) without any human annotation of visual instruction data.

LLaVA 1.5 improved on the original with three targeted changes — an MLP projection, higher-resolution CLIP (336px), and academic VQA data — achieving state-of-the-art on 11 benchmarks while training on a single 8-A100 node in ~1 day.

Model configurations:

	LLaVA 1.0 (13B)	LLaVA 1.5 (13B)
Vision Encoder	CLIP ViT-L/14 (224px)	CLIP ViT-L/14 (336px)
Visual Tokens	16 x 16 = 256	24 x 24 = 576
Projection	Linear (1024 to 5120)	2-layer MLP (1024 to 5120)
LLM	Vicuna-13B v1.3	Vicuna-13B v1.5
Stage 1 Data	595K CC3M filtered	558K LAION-CC-SBU
Stage 2 Data	158K LLaVA-Instruct	665K mixed
Training	8x A100, ~14 hours	8x A100, ~26 hours

LLaVA demonstrated that the data and training recipe matter more than architectural complexity — a lesson that influenced subsequent multimodal models including LLaVA-NeXT, InternVL, and Qwen-VL.

Chapter Roadmap

Click any topic to jump in

Interactive demo

Visual Tokenization

CLIP ViT-L/14 encodes a 336-px image into 576 visual tokens — the raw substrate for multimodal reasoning.

feeds projected tokens to

Vision-Language Projection

A single MLP maps 1024-d CLIP features into the 4096-d LLM embedding space, the sole bridge between modalities.

Multimodal Input

Projected visual tokens are concatenated with text tokens — no architectural change to the transformer.

GPT-4 Instruction Data

GPT-4 distills captions and bounding boxes into 158K multimodal conversations, descriptions, and reasoning samples.

Two-Stage Training

Stage 1 aligns the projector on captions; Stage 2 jointly tunes projector + LLM on instruction data.

unlocked by data + projector

LLaVA 1.5

Higher resolution (336²), MLP projector, and VQA data push LLaVA to SOTA on 11 benchmarks with ~1.2M samples.

Benchmarks & Ablations

85.1% relative GPT-4 score shows data quality and a minimal architecture beat raw scale.

Concept by concept

7 sections, in reading order.

01 / 07Visual Tokenization (CLIP Visual Encoder)

LLaVA converts images into a sequence of visual tokens using a frozen CLIP vision encoder. These tokens are then projected into the language model's embedding space, allowing the LLM to "see" images as naturally as it reads text.

The Problem

Large language models process sequences of discrete text tokens, but images are continuous 2D grids of pixels. To build a multimodal model, we need to convert images into a format the LLM can process:

Images are high-dimensional: A 336x336 RGB image has $336 \times 336 \times 3 = 338,688$ values — far too many to feed directly into an LLM
Semantic gap: Raw pixels don't carry semantic meaning. The color value at pixel (100, 200) tells the LLM nothing about objects, scenes, or concepts in the image
Training from scratch is prohibitive: Learning visual representations from image-text pairs requires billions of training examples (CLIP was trained on 400M pairs) and massive compute

Previous approaches used complex bridging architectures:

Flamingo: Gated cross-attention layers interleaved with LLM layers — expensive and tightly coupled
BLIP-2: Q-Former with 32 learned queries + cross-attention to frozen vision features — complex, requires pre-training the Q-Former itself
These approaches add significant architectural complexity and training overhead, and their cross-attention mechanisms make them harder to scale

The Solution

LLaVA's approach is radically simple: use CLIP ViT-L/14 as a frozen feature extractor, take the grid features (not the [CLS] token), and project them into the LLM's word embedding space.

Step 1: Image to patches

The image is divided into non-overlapping patches:

LLaVA 1.0: Input $224 \times 224$ image, patch size $14 \times 14$ pixels
- Grid: $224/14 = 16$ patches per side
- Total: $16 \times 16 = 256$ visual tokens
- Each patch covers $14 \times 14 = 196$ pixels
LLaVA 1.5: Input $336 \times 336$ image, patch size $14 \times 14$ pixels
- Grid: $336/14 = 24$ patches per side
- Total: $24 \times 24 = 576$ visual tokens
- 2.25x more visual tokens = finer spatial detail

Step 2: Patch embedding

Each $14 \times 14 \times 3$ patch is linearly embedded into a $d_{clip} = 1024$ -dimensional vector:

$z_i = \text{Linear}(\text{flatten}(\text{patch}_i)) + PE_i \quad \text{where } z_i \in \mathbb{R}^{1024}$

A learnable [CLS] token is prepended, and positional embeddings are added.

Step 3: ViT Transformer processing

The patch tokens pass through 24 Transformer layers (ViT-L has 24 layers, 16 heads, $d = 1024$ ):

$Z_{out} = \text{ViT}(z_1, z_2, ..., z_n) \in \mathbb{R}^{(n+1) \times 1024}$

Critical choice: LLaVA uses features from the second-to-last layer (layer 23 of 24), not the final layer. The final layer of CLIP is trained with a contrastive loss that produces abstract, global features optimized for image-text matching. The penultimate layer retains more localized, spatial information — individual objects, textures, spatial relationships — which is more useful for visual question answering.

Step 4: Grid features output

The [CLS] token is discarded. The remaining grid features become the visual token sequence:

LLaVA 1.0: $Z_v \in \mathbb{R}^{256 \times 1024}$ (256 tokens, each 1024-dim)
LLaVA 1.5: $Z_v \in \mathbb{R}^{576 \times 1024}$ (576 tokens, each 1024-dim)

These visual tokens are now ready for projection into the LLM's embedding space. Each token represents a $14 \times 14$ pixel region of the image and carries rich semantic features learned from CLIP's 400M image-text training pairs.

Why CLIP specifically?

CLIP's vision encoder is trained with a contrastive language-image objective, so its features are already aligned with language semantics. A dog patch produces features that are close (in CLIP space) to the word "dog" — this makes the projection layer's job much easier than starting from a vision-only encoder like ImageNet-trained ViTs.

Key Points

1
CLIP ViT-L/14 is kept completely frozen during both training stages — no gradients flow back to the vision encoder. This preserves CLIP's robust visual representations learned from 400M image-text pairs
2
Grid features, not [CLS]: LLaVA uses all $n$ spatial tokens ( $256$ or $576$ ), discarding the global [CLS] token. This preserves spatial layout — the LLM knows where things are, not just what the image contains
3
Penultimate layer features (layer 23/24) retain localized spatial properties. The final layer is too abstract (optimized for global contrastive matching) — ablations showed a 0.96% accuracy drop using final-layer features on ScienceQA
4
Resolution matters: Moving from 224px (256 tokens) to 336px (576 tokens) in LLaVA 1.5 improved performance across all benchmarks by capturing finer details — especially for text-heavy and detail-oriented visual questions
5
Each visual token represents a $14 \times 14$ pixel patch ( $\approx$ 4-6% of image width) — roughly the granularity of a single word in text. This makes the image-as-tokens metaphor surprisingly apt

Mathematical Formulation

Patch Embedding

$z_i = W_e \cdot \text{flatten}(x_p^i) + E_{pos}^i \quad \text{where } W_e \in \mathbb{R}^{1024 \times 588}, \; x_p^i \in \mathbb{R}^{14 \times 14 \times 3}$

Each 14×14×3 = 588-dimensional flattened patch is projected to 1024 dimensions. Positional embedding E_pos is added to encode spatial location in the grid.

Visual Feature Extraction

$Z_v = \text{ViT}_{L-1}(x_{img}) \in \mathbb{R}^{n \times 1024} \quad \text{where } n = (H/14)^2$

ViT_{L-1} denotes the penultimate layer output. For 224px: n=256, for 336px: n=576. These 1024-dim features carry both semantic and spatial information from CLIP pre-training.

Mathematical Intuition

A $224\times 224$ image is cut into $14\times 14 = 196$ non-overlapping $16\times 16$ patches. Each patch is flattened to a vector in $\mathbb{R}^{768}$ and linearly projected, yielding a grid of visual tokens that the frozen CLIP ViT-L/14 encoder refines into rich semantic features $\mathbf{Z}_v \in \mathbb{R}^{196 \times 1024}$ .

The projection layer is LLaVA's key architectural contribution — a simple linear layer (v1.0) or 2-layer MLP (v1.5) that bridges the 1024-dimensional CLIP visual feature space to the 5120-dimensional LLM word embedding space, allowing visual tokens to be processed as if they were language tokens.

The Problem

CLIP's visual features and the LLM's word embeddings live in completely different vector spaces:

CLIP visual features: $Z_v \in \mathbb{R}^{n \times 1024}$ — optimized for contrastive image-text matching. High similarity between "dog" and an image of a dog, but the actual vector values don't correspond to any language model's vocabulary
LLM word embeddings: $H_w \in \mathbb{R}^{m \times 5120}$ (for Vicuna-13B) — trained for next-token prediction. The embedding for "dog" is a 5120-dim vector with statistics learned from billions of text tokens

These spaces differ in:

Dimensionality: 1024 vs 5120 (5x difference)
Training objective: Contrastive matching vs autoregressive prediction
Statistical properties: Different mean, variance, and correlation structures
Semantic organization: Similar concepts may cluster differently in each space

Previous approaches addressed this with heavyweight modules:

Q-Former (BLIP-2): 188M parameters, trained in 3 stages
Gated cross-attention (Flamingo): Interleaves visual cross-attention within every LLM layer, adding ~1.4B parameters for 80B models
Perceiver Resampler (Flamingo): Reduces visual tokens to fixed 64 via learned queries

LLaVA's question: Can we bridge these spaces with something much simpler?

The Solution

LLaVA 1.0: Simple Linear Projection

A single trainable matrix $W$ maps CLIP features into the LLM embedding space:

$H_v = Z_v \cdot W \quad \text{where } W \in \mathbb{R}^{1024 \times 5120}$

Input: $Z_v \in \mathbb{R}^{256 \times 1024}$ (256 visual tokens from CLIP)
Output: $H_v \in \mathbb{R}^{256 \times 5120}$ (256 visual tokens in LLM space)
Parameters: $1024 \times 5120 = 5,242,880$ ( $\approx$ 5M) — just 0.04% of the 13B LLM

This is shockingly lightweight compared to Q-Former's 188M parameters. The authors chose this deliberately: "A simple linear layer is lightweight, which allows us to iterate data-centric experiments quickly."

LLaVA 1.5: 2-Layer MLP with GELU

The linear projection was upgraded to a 2-layer MLP:

$H_v = \text{GELU}(Z_v W_1 + b_1) W_2 + b_2$

where:

$W_1 \in \mathbb{R}^{1024 \times 4096}$ — first projection
$W_2 \in \mathbb{R}^{4096 \times 5120}$ — second projection
GELU activation between layers adds non-linearity

Parameters: $(1024 \times 4096 + 4096) + (4096 \times 5120 + 5120) \approx$ 25M — still tiny compared to Q-Former

Why MLP > Linear?

The linear projection performs a pure affine transformation — it can scale, rotate, and shift the features but cannot learn non-linear mappings between the two spaces. The MLP with GELU activation can:

Disentangle entangled features: CLIP may bundle "color" and "shape" in ways that don't match LLM expectations
Threshold and gate: GELU can suppress weak features (noise) while amplifying strong signals
Learn feature interactions: The hidden layer allows cross-feature computation before final projection

Ablations in LLaVA 1.5 showed the MLP improved MME score by +51.2 over the linear projection.

How projected visual tokens integrate with text:

After projection, visual tokens $H_v$ and text token embeddings $H_w$ are simply concatenated into a single sequence:

$H_{input} = [H_v^1, H_v^2, ..., H_v^n, H_w^1, H_w^2, ..., H_w^m]$

The LLM processes this mixed sequence with standard causal self-attention — no special cross-attention or gating needed. The visual tokens participate in attention exactly like word tokens. This works because the projection layer has mapped them into the same embedding space.

Concrete example (LLaVA 1.5, 13B):

576 visual tokens (each 5120-dim) from a 336px image
~50 text tokens (each 5120-dim) from "Describe this image in detail."
Total input sequence: 626 tokens × 5120 dimensions
The LLM's self-attention computes a $626 \times 626$ attention matrix per head

Key Points

1
Linear projection (5M params) vs Q-Former (188M) vs gated cross-attention (~1.4B): LLaVA proves that simpler connectors work when the vision encoder (CLIP) already produces language-aligned features
2
The MLP projection in LLaVA 1.5 adds a hidden layer ( $1024 \to 4096 \to 5120$ ) with GELU non-linearity, improving MME by +51.2 over linear — the non-linearity helps bridge distributional differences between CLIP and LLM spaces
3
No learned queries or token reduction: Unlike Q-Former (32 queries) or Perceiver (64 queries), LLaVA keeps all visual tokens. This preserves full spatial resolution but increases the input sequence length
4
After projection, visual and text tokens are concatenated into a flat sequence — the LLM processes them with standard self-attention. No architectural changes to the LLM are needed
5
The projection layer is trained first in Stage 1 (frozen LLM, frozen CLIP) to learn the alignment, then fine-tuned alongside the LLM in Stage 2

Mathematical Formulation

Linear Projection (LLaVA 1.0)

$H_v = Z_v W \quad \text{where } W \in \mathbb{R}^{1024 \times 5120}, \; Z_v \in \mathbb{R}^{256 \times 1024}, \; H_v \in \mathbb{R}^{256 \times 5120}$

5.2M parameters. Maps each 1024-dim CLIP token to the LLM's 5120-dim embedding space via matrix multiplication. No non-linearity.

MLP Projection (LLaVA 1.5)

$H_v = \text{GELU}(Z_v W_1 + b_1)W_2 + b_2 \quad \text{where } W_1 \in \mathbb{R}^{1024 \times 4096}, \; W_2 \in \mathbb{R}^{4096 \times 5120}$

~25M parameters. The intermediate 4096-dim hidden layer with GELU lets the projection learn non-linear feature transformations. Output H_v ∈ ℝ^(576×5120) for 336px images.

Mathematical Intuition

The projection layer $f_\phi: \mathbb{R}^{1024} \to \mathbb{R}^{4096}$ is the sole bridge between visual and language spaces. Going from a single linear map (LLaVA 1.0) to a 2-layer MLP with GELU (LLaVA 1.5) adds one hidden layer of $\sim 16$ M parameters, and that tiny change lifts benchmark scores by several points — evidence that more expressive modality bridging matters more than raw parameter count.

LLaVA's training is split into two distinct stages: (1) vision-language alignment pre-training that teaches the projection layer to map visual features into the language space, and (2) end-to-end instruction fine-tuning that teaches the model to follow visual instructions. This separation is critical — it prevents catastrophic forgetting of the LLM's language capabilities while efficiently learning visual understanding.

The Problem

Training a multimodal model end-to-end from scratch faces multiple challenges:

Catastrophic forgetting: If you immediately fine-tune the LLM on visual data, it can lose its text-only capabilities. The model might learn to describe images but forget how to follow complex instructions, reason logically, or maintain conversational coherence
Misaligned feature spaces: At initialization, the projection layer outputs random vectors in the LLM's embedding space. If the LLM is also updating its weights, it must simultaneously (a) learn to interpret these noisy visual tokens and (b) learn to generate appropriate responses — two competing objectives
Data efficiency: High-quality instruction-following data is expensive to create. Using it before the model can even parse visual inputs wastes these valuable samples on a model that can't yet "see"
Compute cost: Training all parameters simultaneously on all data is expensive. It's more efficient to first solve the simpler alignment problem with cheap data, then solve the harder instruction-following problem

The analogy: It's like teaching a student a foreign language. Stage 1 is vocabulary drills (map words to meanings). Stage 2 is conversation practice (use vocabulary in context). Skipping vocabulary drills and jumping to conversations doesn't work.

The Solution

Stage 1: Vision-Language Alignment Pre-training

Goal: Train the projection layer to produce visual token embeddings that the LLM can interpret.

What's frozen vs trained:

Frozen: CLIP ViT-L/14 (vision encoder) + LLM (all parameters)
Trained: Projection layer only (5M params for linear, 25M for MLP)

Data: Image-caption pairs converted to single-turn conversations

LLaVA 1.0: 595K filtered pairs from CC3M (Conceptual Captions)
LLaVA 1.5: 558K pairs from LAION-CC-SBU with BLIP-generated captions

Format: Each caption is reformulated as an instruction:

Human: <image>
Describe the image concisely.
Assistant: A dog playing fetch in a park with green grass...

Training details (LLaVA 1.0):

Learning rate: $2 \times 10^{-3}$ (relatively high — only the projection layer updates)
Batch size: 128
Epochs: 1
Duration: ~4 hours on 8x A100 GPUs
Optimizer: Adam with cosine LR schedule

Training details (LLaVA 1.5):

Learning rate: $1 \times 10^{-3}$ (halved due to MLP having more parameters)
Batch size: 256
Epochs: 1
Warmup ratio: 0.03

What the model learns: After Stage 1, the projection layer converts CLIP's visual features into token embeddings that the LLM treats as meaningful language tokens. The visual tokens for a dog image will land near the word embeddings for "dog," "animal," "pet" in the LLM's space. The LLM can now generate basic captions — but it cannot yet have conversations, answer complex questions, or reason about images.

Stage 2: End-to-End Visual Instruction Tuning

Goal: Teach the model to follow visual instructions — describe images, answer questions, reason about visual content, engage in multi-turn conversations.

What's frozen vs trained:

Frozen: CLIP ViT-L/14 (still frozen)
Trained: Projection layer + entire LLM (both unfrozen)

Data (LLaVA 1.0): 158K instruction-following samples (GPT-4 generated):

58K multi-turn conversations
23K detailed image descriptions
77K complex visual reasoning

Data (LLaVA 1.5): 665K mixed samples:

158K LLaVA-Instruct (original)
40K ShareGPT (text-only conversations to maintain language skills)
83K VQAv2 + 72K GQA + 9K OKVQA (visual QA)
80K OCRVQA + 22K TextCaps (OCR/text recognition)
48K RefCOCO (referring expressions)
86K Visual Genome (region descriptions)

Training details (LLaVA 1.0):

Learning rate: $2 \times 10^{-5}$ (100x lower than Stage 1 — gentle fine-tuning)
Batch size: 32
Epochs: 3
Duration: ~10 hours on 8x A100 GPUs
Precision: BF16/TF32, FSDP + gradient checkpointing

Training details (LLaVA 1.5):

Learning rate: $2 \times 10^{-5}$
Batch size: 128
Optimizer: AdamW (weight decay = 0)
Warmup ratio: 0.03
Duration: ~20 hours on 8x A100 GPUs

Why the LR difference matters: Stage 1 uses $2 \times 10^{-3}$ because only the randomly-initialized projection layer is updating — it needs large steps to learn quickly. Stage 2 uses $2 \times 10^{-5}$ (100x smaller) because the pre-trained LLM is now updating — large steps would destroy its existing language capabilities.

Ablation: Training without Stage 1 (end-to-end from scratch) drops ScienceQA accuracy from 90.92% to 85.81% — a 5.11% drop, confirming that pre-alignment is essential.

Key Points

1
Stage 1 trains only the projection (~5-25M params) with frozen CLIP + LLM, using cheap caption data. Stage 2 unfreezes the LLM and trains on instruction data — this ordering prevents catastrophic forgetting
2
Learning rate gap is 100x: Stage 1 uses $2 \times 10^{-3}$ (fast projection learning), Stage 2 uses $2 \times 10^{-5}$ (gentle LLM fine-tuning). This asymmetry is critical for preserving the LLM's pre-trained knowledge
3
Vision encoder stays frozen throughout both stages — CLIP's features are already high-quality from 400M image-text pairs. Fine-tuning it would risk overfitting on the much smaller LLaVA dataset
4
Skipping Stage 1 costs 5.11% on ScienceQA (85.81% vs 90.92%). The projection layer needs to learn a reasonable mapping before the LLM can meaningfully learn from visual instructions
5
LLaVA 1.5 adds academic VQA data in Stage 2 (665K vs 158K samples), including structured QA from VQAv2, GQA, and OCR tasks — this significantly improves factual accuracy and reduces hallucination

Mathematical Formulation

Stage 1 Loss (Caption Prediction)

$\mathcal{L}_1 = -\sum_{i=1}^{T} \log p_\theta(w_i \mid Z_v, w_{<i}) \quad \text{where } \theta = \{W\} \text{ only}$

Standard autoregressive cross-entropy loss, but gradients only update the projection W (or MLP). The LLM generates captions conditioned on projected visual tokens, but its weights don't change.

Stage 2 Loss (Instruction Following)

$\mathcal{L}_2 = -\sum_{i=1}^{T} \log p_\theta(x_i \mid Z_v, X_{instruct}, x_{<i}) \quad \text{where } \theta = \{W, \phi_{LLM}\}$

Same loss form, but now both the projection and the full LLM update. Only assistant response tokens contribute to the loss — human instruction tokens are masked.

Mathematical Intuition

Stage 1 freezes $\theta_{\text{CLIP}}$ and $\theta_{\text{LLM}}$ and trains only the projector $\phi$ on 558K image-caption pairs — a warm-up that aligns visual features to word embedding space. Stage 2 unfreezes the LLM and fine-tunes $\{\phi, \theta_{\text{LLM}}\}$ jointly on 158K instruction pairs. Decoupling alignment from instruction tuning prevents catastrophic forgetting of the LLM's pretraining.

Architecture

Data flow through the complete pipeline:

Image ( $336 \times 336 \times 3$ ) enters frozen CLIP ViT-L/14
CLIP processes 576 patches through 24 Transformer layers → $Z_v \in \mathbb{R}^{576 \times 1024}$
MLP Projection maps to LLM space → $H_v \in \mathbb{R}^{576 \times 5120}$
Text tokens are embedded → $H_w \in \mathbb{R}^{m \times 5120}$
Concatenation: $[H_v; H_w] \in \mathbb{R}^{(576+m) \times 5120}$
Vicuna-13B processes with causal self-attention (40 layers, 40 heads)
Output head predicts next token: $\mathbb{R}^{5120} \to \mathbb{R}^{32000}$ (vocabulary)

Stage 1: Gradients flow from output → through LLM (no updates) → through projection (updates) → stop at CLIP Stage 2: Gradients flow from output → through LLM (updates) → through projection (updates) → stop at CLIP

LLaVA's most creative contribution is using language-only GPT-4 to generate multimodal instruction-following data. By providing GPT-4 with image captions and bounding box annotations (text-only), the authors created 158K high-quality visual conversation, description, and reasoning samples — without GPT-4 ever seeing the actual images.

The Problem

Instruction-following data is the key ingredient for turning a base LLM into a helpful assistant (as shown by InstructGPT, ChatGPT, etc.). For multimodal models, we need instruction data that teaches the model to:

Describe images in detail when asked
Answer specific questions about image content
Reason about complex visual scenes (spatial relationships, counting, cause-and-effect)
Engage in multi-turn conversations about images

The data bottleneck:

Human annotation is expensive: Collecting "describe this image" + "ask a complex question" + "provide a detailed answer" triplets at scale requires trained annotators. At ~ $0.50/sample, 158K samples would cost ~$ 79K
Existing datasets are limited: VQA datasets have short, factual answers ("yes/no", "3", "red"). They don't teach conversational ability, detailed description, or multi-step reasoning
GPT-4V didn't exist yet: In early 2023, GPT-4 could only process text. There was no API for generating visual instruction data directly from images

The question: Can we use a text-only model to create multimodal training data?

The Solution

LLaVA's insight: Images can be described to GPT-4 using text — captions and bounding boxes carry enough information for GPT-4 to generate realistic visual conversations.

Input to GPT-4 (text only):

For each COCO image, GPT-4 receives:

5 human-written captions (from COCO annotations)
Bounding boxes in the format: [object_name]: [x_min, y_min, x_max, y_max] (normalized coordinates)

GPT-4 never sees the actual image — it works entirely from these text descriptions.

Three types of instruction data generated:

1. Multi-turn Conversations (58K samples)

GPT-4 generates Q&A pairs that a curious person might ask when looking at the image:

Object identification: "What objects are in the scene?"
Counting: "How many people are sitting at the table?"
Spatial reasoning: "What is to the left of the red car?"
Action recognition: "What is the woman doing?"
Object attributes: "What color is the dog's collar?"

Format: Multiple rounds of Human/Assistant dialogue, 3-8 turns per image.

2. Detailed Descriptions (23K samples)

GPT-4 generates comprehensive image narrations covering:

Main subjects and their attributes
Background and setting
Actions and interactions
Spatial layout and composition
Mood, lighting, and atmosphere

These are typically 3-5 paragraphs, much more detailed than standard captions.

3. Complex Reasoning (77K samples)

GPT-4 generates questions requiring multi-step reasoning:

"Why might the person be carrying an umbrella?" (weather inference)
"What event is likely happening?" (scene understanding)
"Based on the objects, what room is this?" (contextual reasoning)
"What would happen if..." (counterfactual reasoning)

Each answer includes step-by-step reasoning chains.

Prompt engineering:

The authors used carefully designed seed examples (in-context learning) for each category. Each seed shows GPT-4:

The format of captions + bounding boxes
The style of desired output (how to phrase questions, how detailed answers should be)
Guidelines: don't mention "the caption says" — pretend you're looking at the image

Quality validation:

The authors manually inspected random samples and found GPT-4's outputs were:

Factually consistent with the captions/boxes (rarely hallucinated objects not in descriptions)
Diverse in question types (not repetitive)
Detailed and helpful in answers
Natural in conversational flow

Total dataset: 158K samples (LLaVA-Instruct-158K)

58K conversation + 23K detailed description + 77K complex reasoning
All generated from ~80K unique COCO images
Generation cost: ~$500 in GPT-4 API calls (April 2023 pricing)
Equivalent human annotation cost: ~$79K (150x cheaper)

Key Points

1
GPT-4 never sees the image — it receives 5 captions + bounding boxes as text. Despite this, it generates high-quality visual conversations, detailed descriptions, and complex reasoning chains
2
Three data categories serve different training purposes: conversations teach multi-turn dialogue, descriptions teach visual detail, reasoning teaches inference. The mix matters — conversation-only training drops performance by 11.3%
3
Cost efficiency: ~ $500 in GPT-4 API calls vs ~$ 79K for equivalent human annotation (150x cheaper). This made multimodal instruction tuning accessible to academic labs
4
The data generation approach has been widely adopted — InstructBLIP, ShareGPT4V, and LLaVA-NeXT all build on this paradigm of using strong LLMs to generate training data
5
Limitation: Since GPT-4 works from captions, it cannot generate data about visual details not captured in the captions — fine-grained textures, exact spatial relationships, or subtle visual attributes may be missing

Mathematical Intuition

Because GPT-4 is text-only, images are distilled into two structured text prompts: dense captions and bounding box lists. GPT-4 then produces three data types — conversations ( $\approx 58$ K), detailed descriptions ( $\approx 23$ K), and complex reasoning ( $\approx 77$ K) — a diversity that seeds emergent multimodal reasoning despite never showing GPT-4 the pixels.

Comparison

Before / Traditional

Traditional VQA Datasets:

Short factual answers ("yes", "3", "blue")
Single-turn questions only
No reasoning chains or explanations
Human-annotated at ~$0.50/sample
Limited to pre-defined question types
Examples: VQAv2, GQA, Visual7W

After / This Paper

LLaVA-Instruct-158K (GPT-4 generated):

Detailed, conversational responses (3-5 paragraphs)
Multi-turn dialogues (3-8 rounds)
Step-by-step reasoning chains
~$0.003/sample via GPT-4 API
Open-ended question types
Covers description, reasoning, conversation

LLaVA treats visual and text tokens as a single unified sequence, feeding them together through the LLM's standard causal self-attention. This elegantly simple approach — concatenating projected image tokens with text token embeddings — avoids any architectural modifications to the language model.

The Problem

Once we have visual tokens (from CLIP + projection) and text tokens (from the tokenizer), we need to decide how to feed them to the LLM. This raises several design questions:

Ordering: Should image tokens come before or after text tokens? Does it matter?
Attention pattern: Should visual tokens attend to text tokens and vice versa? Should there be asymmetric attention patterns?
Multi-turn handling: In a conversation with multiple turns, should the image be re-encoded for each turn?
Loss computation: When training, which tokens should contribute to the loss? All tokens or only response tokens?

Different approaches in the literature:

Flamingo: Image tokens are processed separately via cross-attention — text tokens query image features at each layer but image tokens don't see text
BLIP-2: Q-Former first queries image features, then outputs 32 fixed tokens that are prepended to text
These approaches require custom attention patterns or separate processing streams

The Solution

LLaVA's approach: Simple concatenation

Visual and text tokens form a single flat sequence:

$X_{input} = [v_1, v_2, ..., v_n, w_1, w_2, ..., w_m]$

where $v_i \in \mathbb{R}^{5120}$ are projected visual tokens and $w_j \in \mathbb{R}^{5120}$ are text embeddings.

Token layout for a single-turn example:

[IMG_1][IMG_2]...[IMG_576][Human:][Describe][this][image][.][Assistant:]...[<STOP>]
|←— 576 visual tokens —→|←——— instruction tokens ———→|←— response tokens —→|

Total sequence length (LLaVA 1.5): 576 (image) + ~50-200 (instruction) + ~100-500 (response) = ~750-1300 tokens

Attention pattern: Standard causal self-attention

Every token attends to all previous tokens (including all visual tokens):

Visual tokens attend to all preceding visual tokens
Text tokens attend to all visual tokens + all preceding text tokens
Response tokens attend to everything before them

This means the LLM can:

Use visual context when generating each response word
Relate text concepts to specific visual tokens (spatial grounding)
No special cross-attention mechanism needed

Multi-turn handling:

The image tokens are included only in the first turn. Subsequent conversation turns are text-only:

Turn 1: [IMG_1..576][Human: What's in this image?][Assistant: ...]
Turn 2: [Human: How many dogs are there?][Assistant: ...]
Turn 3: [Human: What color is the largest one?][Assistant: ...]

Because of causal attention, all later turns can still attend to the image tokens from turn 1 — the visual context persists throughout the conversation without re-encoding.

Loss masking:

$\mathcal{L} = -\sum_{i \in \text{response\_tokens}} \log p(x_i \mid x_{<i})$

Only assistant response tokens contribute to the loss. Tokens from:

Image features: masked (no gradient through prediction)
Human instructions: masked (input, not prediction target)
System message: masked

This ensures the model learns to generate helpful responses, not to predict the question or reproduce image features.

Special tokens:

<STOP> token (mapped to ###) marks the end of each assistant response
System message follows Vicuna-v0 format: "A chat between a curious human and an artificial intelligence assistant..."
Image placement: randomly before or after the first instruction in training (data augmentation)

Key Points

1
Flat concatenation: Visual and text tokens form one sequence — the LLM processes them with standard causal self-attention. Zero architectural changes to the language model
2
Visual tokens always come first (in the final model). The LLM builds visual context before processing the instruction. Image placement was randomized during training for robustness
3
Multi-turn efficiency: The image is encoded once in turn 1 and persists in the KV-cache. All subsequent turns attend to these cached visual tokens without re-encoding
4
Response-only loss: Only assistant tokens contribute to training loss — image tokens and human instruction tokens are masked. This focuses learning on generating helpful responses, not reproducing inputs
5
Sequence length is the bottleneck: 576 visual tokens (LLaVA 1.5) consume over half the typical context, limiting the length of conversations. LLaVA-NeXT addresses this with dynamic resolution and token merging

Mathematical Formulation

Autoregressive Loss with Token Masking

$p(X_a \mid X_v, X_{instruct}) = \prod_{i=1}^{L} p_\theta(x_i \mid X_v, X_{instruct,<i}, X_{a,<i})$

X_v = visual tokens, X_instruct = instruction tokens (masked from loss), X_a = assistant response tokens (loss computed here). The model predicts each response token conditioned on all preceding visual, instruction, and response tokens.

Mathematical Intuition

Visual tokens and text tokens share the same $4096$ -dimensional embedding space after projection, so the LLM sees a single flat sequence $[\mathbf{H}_v; \mathbf{H}_q]$ . The 196 visual tokens act as a soft prompt; no architectural changes to the transformer are needed — image understanding emerges purely from concatenation and next-token prediction.

LLaVA 1.5 demonstrated that three simple, well-motivated changes — an MLP projection layer, higher-resolution visual encoding, and academic task-oriented training data — are sufficient to achieve state-of-the-art across 11 benchmarks. The paper's subtitle "Improved Baselines with Visual Instruction Tuning" reflects its philosophy: strong baselines matter more than architectural novelty.

The Problem

After LLaVA 1.0 established the basic recipe, several gaps remained:

Resolution limitation: 224px images lose fine details — small text, distant objects, and subtle visual patterns are unrecoverable from $16 \times 16 = 256$ tokens
Linear projection ceiling: The single-matrix projection cannot learn non-linear feature transformations, limiting alignment quality between CLIP and LLM spaces
Data bias: Training only on GPT-4-generated conversations creates a bias toward descriptive, open-ended responses. The model struggles with factual visual questions (exact counts, text reading, specific object attributes) that require precise answers
Hallucination: LLaVA 1.0 tends to hallucinate objects not present in images — a problem exacerbated by GPT-4 training data that sometimes describes objects inferred from captions rather than actually visible

The research question: How much can we improve by making targeted changes to the baseline, without fundamentally altering the architecture?

The Solution

LLaVA 1.5 makes exactly three changes:

Change 1: MLP Projection (Linear → 2-layer MLP)

$\text{Linear}: H_v = Z_v W \quad\to\quad \text{MLP}: H_v = \text{GELU}(Z_v W_1)W_2$

$W_1 \in \mathbb{R}^{1024 \times 4096}$ , $W_2 \in \mathbb{R}^{4096 \times 5120}$

Impact: MME score +51.2 (from 1272.6 to 1323.8 with response formatting alone)

Change 2: Higher Resolution (224px → 336px)

	LLaVA 1.0	LLaVA 1.5
Input resolution	224 × 224	336 × 336
Patches per side	16	24
Total visual tokens	256	576
Pixels per patch	196	196 (same)
Effective detail	~14px per token	~14px per token

The higher resolution doesn't change the patch size — it simply provides more patches covering the image at the same per-patch granularity. The result is 2.25x more visual tokens, capturing finer spatial details.

Impact: ~2-3% improvement across benchmarks, especially on text-heavy tasks (TextVQA, OCRVQA).

Change 3: Academic Task Data (158K → 665K)

LLaVA 1.5 augments the GPT-4 instruction data with academic VQA datasets, formatted as instruction-following conversations:

Dataset	Samples	Purpose
LLaVA-Instruct	158K	Conversation, description, reasoning
ShareGPT	40K	Text-only dialogues (maintain language)
VQAv2	83K	Visual question answering
GQA	72K	Compositional spatial reasoning
OKVQA	9K	Knowledge-requiring VQA
OCRVQA	80K	OCR-based document QA
TextCaps	22K	Image captioning with text reading
A-OKVQA	66K	Augmented knowledge VQA
RefCOCO	48K	Referring expression comprehension
Visual Genome	86K	Region descriptions

Response formatting: Academic datasets originally have short answers ("yes", "3"). LLaVA 1.5 adds simple response formatting prompts: "Answer the question using a single word or phrase." This teaches the model when to be concise vs. detailed.

Impact: Response formatting alone improved MME from 809.6 to 1323.8 (+514.2).

Combined results (LLaVA-1.5-13B):

Benchmark	LLaVA 1.0	LLaVA 1.5	Best Competitor
VQAv2	76.6	80.0	78.2 (InstructBLIP)
GQA	—	63.3	49.5 (InstructBLIP)
TextVQA	—	61.3	50.7 (InstructBLIP)
MME	—	1531.3	1580.7 (Qwen-VL-Chat)
POPE	—	86.2	78.3 (Shikra)
MM-Vet	—	36.1	34.4 (InstructBLIP)

LLaVA 1.5 achieves SOTA or near-SOTA on 11/12 benchmarks — using only publicly available data and training on a single 8-A100 node in ~1 day.

Key Points

1
Three targeted changes — MLP projection, 336px resolution, academic VQA data — achieved SOTA on 11 benchmarks. No new architecture, no new pre-training, no proprietary data
2
Response formatting is surprisingly powerful: Simply adding "Answer with a single word or phrase" improved MME by +514.2. The model was capable but didn't know when to be concise vs. detailed
3
MLP > Linear by +51.2 MME: The non-linear projection layer better bridges the distributional gap between CLIP features and LLM embeddings. Two layers are sufficient — no need for deeper MLPs
4
Academic VQA data reduces hallucination: Training on factual QA datasets (VQAv2, GQA) teaches the model to answer precisely rather than generating plausible-sounding but incorrect descriptions. POPE hallucination score improved from ~75% to 86.2%
5
Reproducibility was a design goal: All data is public, code is open-source, training fits on one 8-A100 node. This democratized multimodal LLM research — dozens of follow-up works built directly on LLaVA 1.5

Mathematical Formulation

Visual Token Count

$n_{tokens} = \left(\frac{H}{p}\right) \times \left(\frac{W}{p}\right) = \left(\frac{336}{14}\right)^2 = 576$

H, W = image resolution, p = patch size. Higher resolution with the same patch size gives more tokens and finer spatial detail. Cost: attention is O(n²), so 576 tokens costs ~5x more attention compute than 256 tokens.

Mathematical Intuition

Raising input resolution from $224^2$ to $336^2$ increases patch count from $196$ to $576$ (a $2.94\times$ token budget), giving finer spatial detail at the cost of $\mathcal{O}(n^2)$ attention compute. Combined with the MLP projector and academic VQA data, this unlocks state-of-the-art on 11 benchmarks despite using only $\sim 1.2$ M training samples.

Comparison

Before / Traditional

LLaVA 1.0:

Linear projection (5M params)
224px resolution (256 visual tokens)
158K GPT-4 instruction data only
85.1% relative to GPT-4 (LLaVA-Bench)
No academic VQA training
Prone to hallucination on factual questions

After / This Paper

LLaVA 1.5:

MLP projection (25M params)
336px resolution (576 visual tokens)
665K mixed data (instruction + academic VQA)
SOTA on 11 benchmarks
Response formatting for task-appropriate answers
Reduced hallucination (POPE 86.2%)

LLaVA and LLaVA 1.5 were evaluated across a comprehensive suite of benchmarks spanning visual question answering, multimodal reasoning, hallucination detection, and open-ended visual chat. The ablation studies reveal which components contribute most to performance — and some results are surprising.

The Problem

Evaluating multimodal models is inherently difficult because different tasks test different capabilities:

Factual VQA (VQAv2, GQA): Tests precise object recognition, counting, attribute identification — requires concise, accurate answers
Knowledge VQA (OKVQA, A-OKVQA): Requires external knowledge beyond what's visible in the image
OCR tasks (TextVQA, OCRVQA): Tests ability to read text in images — depends heavily on visual resolution
Hallucination (POPE): Asks "Is there a [object] in the image?" — tests whether the model confabulates objects
Open-ended chat (LLaVA-Bench, MM-Vet): Tests conversational ability, reasoning depth, detail — evaluated by GPT-4 as judge
Comprehensive (MME, MMBench, SEED): Broad coverage across perception and cognition tasks

No single metric captures multimodal capability — a model can excel at description but hallucinate objects, or answer VQA correctly but fail at multi-step reasoning.

The Solution

LLaVA 1.0 Results:

LLaVA-Bench (In-the-Wild) — 24 diverse images, 60 questions, GPT-4 as judge (score relative to text-only GPT-4 with ground-truth captions):

Model	Overall	Conversation	Detail	Reasoning
BLIP-2 (13B)	38.1%	54.8%	25.5%	29.8%
OpenFlamingo	19.1%	—	—	—
LLaVA (13B)	67.3%	64.3%	55.9%	81.7%

LLaVA's reasoning score of 81.7% relative to GPT-4 is remarkable — it outperforms GPT-4 on some complex reasoning questions, likely because the visual grounding provides concrete evidence that pure text descriptions miss.

ScienceQA — 21K multimodal science questions across 26 topics:

Model	Accuracy
GPT-4 (text only)	82.69%
MM-CoT Large (prior SOTA)	91.68%
LLaVA alone	90.92%
LLaVA + GPT-4 judge	92.53% (new SOTA)

LLaVA 1.5 Results (comprehensive benchmarks):

Benchmark	LLaVA-1.5-7B	LLaVA-1.5-13B	InstructBLIP-13B	Qwen-VL-Chat
VQAv2	78.5	80.0	—	78.2
GQA	62.0	63.3	49.5	57.5
VisWiz	50.0	53.6	33.4	—
TextVQA	58.2	61.3	50.7	—
MME (perception)	1510.7	1531.3	—	1487.6
POPE	85.9	86.2	—	—
MMBench	64.3	67.7	—	60.6
SEED-Bench	66.1	68.2	—	58.2
MM-Vet	31.1	36.1	25.6	—

Key Ablation Studies:

1. Projection type (ScienceQA):

No pre-training (end-to-end only): 85.81% (-5.11%)
With pre-training + linear projection: 90.92%
With pre-training + MLP projection: improved across all benchmarks

2. Visual features layer:

Final CLIP layer: 89.96%
Penultimate layer: 90.92% (+0.96%) — localized features beat global features

3. Chain-of-thought ordering:

Reasoning first, then answer: 89.77%
Answer first, then reasoning: 90.92% (+1.15%) — generating the answer first provides a "hypothesis" that guides reasoning

4. Model scale:

LLaVA 7B: 89.84%
LLaVA 13B: 90.92% (+1.08%) — bigger LLM helps but with diminishing returns

5. Data composition ablation (LLaVA 1.0):

Conversation + Detail + Reasoning (full): 85.1% (LLaVA-Bench)
Detail + Reasoning only: 81.9% (-3.2%)
Conversation only: 73.8% (-11.3%)
No instruction tuning: 21.5% (-63.6%)

The 21.5% without instruction tuning confirms that Stage 2 is essential — the pre-aligned model can "see" but cannot have useful conversations.

6. Response formatting (LLaVA 1.5):

Without formatting prompts: MME 809.6
With formatting prompts: MME 1323.8 (+514.2)

This massive jump shows the model had latent capability that was unlocked simply by telling it what format to use. The model knew the answer but was generating it in the wrong format (e.g., generating a full sentence when a single word was expected).

Key Points

1
LLaVA achieves 81.7% of GPT-4's reasoning on visual tasks (LLaVA-Bench) while being much smaller (13B vs GPT-4's ~1.8T estimated params) — visual grounding provides concrete evidence that compensates for smaller model capacity
2
Skipping Stage 1 drops accuracy by 5.11% and removing instruction tuning drops it by 63.6% — both stages are essential, but Stage 2 contributes far more to final performance
3
Penultimate CLIP layer > final layer (+0.96%): The final layer is too abstract (global contrastive features). Layer 23 retains spatial, localized information better suited for visual QA
4
Answer-first beats reasoning-first (+1.15% on ScienceQA): Generating the answer as a hypothesis before explaining it produces better reasoning chains than trying to reason to a conclusion
5
Response formatting provides the single largest improvement (+514.2 MME): The model already knew the answers — it just needed to be told whether to respond with a single word or a paragraph. This highlights the importance of prompt engineering in multimodal models

Mathematical Formulation

GPT-4 Evaluation Score

$\text{Score}_{\text{relative}} = \frac{\text{Score}_{\text{model}}}{\text{Score}_{\text{GPT-4}}} \times 100\%$

Both the model's and GPT-4's responses are rated by GPT-4 on a 1-10 scale for helpfulness, relevance, accuracy, and detail. Scores above 100% indicate the multimodal model outperforms text-only GPT-4 with ground-truth captions.

Mathematical Intuition

LLaVA-1.5-13B reaches a relative GPT-4 score of $85.1\%$ on LLaVA-Bench with only $\approx 1.2$ M samples — demonstrating that data quality and architectural minimalism (CLIP + MLP + Vicuna) beat scale. Each ablation isolates one variable, making the improvements statistically attributable rather than coincidental.

Go deeper

Study Plans

VLM Study Plan LLM Study Plan All Papers Practice Problems

Paper Overview

Chapter Roadmap

Visual Tokenization

Vision-Language Projection

Multimodal Input

GPT-4 Instruction Data

Two-Stage Training

LLaVA 1.5

Benchmarks & Ablations

Concept by concept

Visual Tokenization (CLIP Visual Encoder)

The Problem

The Solution

Key Points

Mathematical Formulation

Patch Embedding

Visual Feature Extraction

Mathematical Intuition

Vision-Language Projection

The Problem

The Solution

Key Points

Mathematical Formulation

Linear Projection (LLaVA 1.0)

MLP Projection (LLaVA 1.5)

Mathematical Intuition

Two-Stage Training Pipeline

The Problem

The Solution

Key Points

Mathematical Formulation

Stage 1 Loss (Caption Prediction)

Stage 2 Loss (Instruction Following)

Mathematical Intuition

Architecture

GPT-4 Instruction Data Generation

The Problem

The Solution

Key Points

Mathematical Intuition

Comparison

Before / Traditional

After / This Paper

Multimodal Input Representation

The Problem

The Solution

Key Points

Mathematical Formulation

Autoregressive Loss with Token Masking

Mathematical Intuition

LLaVA 1.5: Improved Baselines

The Problem

The Solution

Key Points

Mathematical Formulation

Visual Token Count

Mathematical Intuition

Comparison

Before / Traditional

After / This Paper

Benchmarks, Ablations & Key Results

The Problem

The Solution

Key Points

Mathematical Formulation

GPT-4 Evaluation Score

Mathematical Intuition

Further Reading

Related Papers

Study Plans