LLaVA: Visual Instruction Tuning

Large language models process sequences of discrete text tokens, but images are continuous 2D grids of pixels. To build a multimodal model, we need to convert images into a format the LLM can process:

• Images are high-dimensional: A 336x336 RGB image has $336 \times 336 \times 3 = 338,688$ values — far too many to feed directly into an LLM
• Semantic gap: Raw pixels don't carry semantic meaning. The color value at pixel (100, 200) tells the LLM nothing about objects, scenes, or concepts in the image
• Training from scratch is prohibitive: Learning visual representations from image-text pairs requires billions of training examples (CLIP was trained on 400M pairs) and massive compute

Previous approaches used complex bridging architectures:

• Flamingo: Gated cross-attention layers interleaved with LLM layers — expensive and tightly coupled
• BLIP-2: Q-Former with 32 learned queries + cross-attention to frozen vision features — complex, requires pre-training the Q-Former itself
• These approaches add significant architectural complexity and training overhead, and their cross-attention mechanisms make them harder to scale

LLaVA's approach is radically simple: use CLIP ViT-L/14 as a frozen feature extractor, take the grid features (not the [CLS] token), and project them into the LLM's word embedding space.

Step 1: Image to patches

The image is divided into non-overlapping patches:

• LLaVA 1.0: Input $224 \times 224$ image, patch size $14 \times 14$ pixels

  • Grid: $224/14 = 16$ patches per side
  • Total: $16 \times 16 = 256$ visual tokens
  • Each patch covers $14 \times 14 = 196$ pixels

• LLaVA 1.5: Input $336 \times 336$ image, patch size $14 \times 14$ pixels

  • Grid: $336/14 = 24$ patches per side
  • Total: $24 \times 24 = 576$ visual tokens
  • 2.25x more visual tokens = finer spatial detail

Step 2: Patch embedding

Each $14 \times 14 \times 3$ patch is linearly embedded into a $d_{clip} = 1024$-dimensional vector:

$z_i = \text{Linear}(\text{flatten}(\text{patch}_i)) + PE_i \quad \text{where } z_i \in \mathbb{R}^{1024}$

A learnable [CLS] token is prepended, and positional embeddings are added.

Step 3: ViT Transformer processing

The patch tokens pass through 24 Transformer layers (ViT-L has 24 layers, 16 heads, $d = 1024$):

$Z_{out} = \text{ViT}(z_1, z_2, ..., z_n) \in \mathbb{R}^{(n+1) \times 1024}$

Critical choice: LLaVA uses features from the second-to-last layer (layer 23 of 24), not the final layer. The final layer of CLIP is trained with a contrastive loss that produces abstract, global features optimized for image-text matching. The penultimate layer retains more localized, spatial information — individual objects, textures, spatial relationships — which is more useful for visual question answering.

Step 4: Grid features output

The [CLS] token is discarded. The remaining grid features become the visual token sequence:

• LLaVA 1.0: $Z_v \in \mathbb{R}^{256 \times 1024}$ (256 tokens, each 1024-dim)
• LLaVA 1.5: $Z_v \in \mathbb{R}^{576 \times 1024}$ (576 tokens, each 1024-dim)

These visual tokens are now ready for projection into the LLM's embedding space. Each token represents a $14 \times 14$ pixel region of the image and carries rich semantic features learned from CLIP's 400M image-text training pairs.

Why CLIP specifically?

CLIP's vision encoder is trained with a contrastive language-image objective, so its features are already aligned with language semantics. A dog patch produces features that are close (in CLIP space) to the word "dog" — this makes the projection layer's job much easier than starting from a vision-only encoder like ImageNet-trained ViTs.