Chapter 4: Vision-Language Alignment
Master the bridge architectures that connect vision encoders to language models. From simple linear projections to sophisticated Q-Former and Perceiver Resampler designs, understand how visual representations are translated into the language model's embedding space and how different approaches trade off between token efficiency, information preservation, and computational cost.
Chapter Overview
The alignment problem in VLMs is fundamentally about bridging two representation spaces that were trained independently: the vision encoder's feature space (optimized for visual similarity via contrastive learning) and the LLM's embedding space (optimized for next-token prediction of text). These spaces encode different types of information in different geometric structures, and the bridge module must translate between them faithfully.
The bridge architecture is arguably the most critical design decision in a VLM. It determines how much visual information reaches the language model, how many tokens the LLM must process (directly impacting inference cost), and whether the model can preserve fine-grained spatial details or must compress to a coarse summary.
The field has explored a rich design space:
- Linear Projection (LLaVA): The simplest bridge---a single matrix multiply. Fast, preserves all spatial tokens, but limited in representational capacity.
- Q-Former (BLIP-2): Learned query tokens cross-attend to image features, compressing 257 image tokens to just 32. Powerful but adds significant parameters and complexity.
- Perceiver Resampler (Flamingo): Fixed latent tokens cross-attend to visual features. A clean, principled approach to information compression.
- MLP Projectors (LLaVA-1.5): Multi-layer projectors with nonlinearity. The practical sweet spot that most modern VLMs use.
- Dynamic Resolution (LLaVA-NeXT, SigLIP2): Handle arbitrary image sizes without distortion.
This chapter covers each approach in detail, with mathematical formulations, implementation walkthroughs, and a systematic comparison of trade-offs.
Chapter Roadmap
Click any topic to jump in
The Alignment Problem
Vision encoders and LLMs speak different languages. CLIP groups by visual similarity (dogs and cats are close), LLMs group by linguistic meaning (dogs and cats are far apart). A bridge module must translate between these two worlds.
Each takes a different approach to the same problem
Linear Projection (LLaVA)
The simplest possible bridge: one matrix multiply converts each image patch independently. Surprisingly competitive — if CLIP and the LLM are both strong, a simple coordinate transform is enough.
Q-Former (BLIP-2)
32 learnable "question" tokens cross-attend to the image, each specializing to extract different info (objects, colors, layout). Compresses 257 image tokens down to just 32 — an 8x reduction.
Perceiver (Flamingo)
Fixed latent tokens iteratively refine themselves by repeatedly cross-attending to image features. Similar goal to Q-Former (compression) but with a cleaner, more principled architecture.
MLP Projectors
A 2-layer MLP (linear → GELU → linear) replaces the single linear projection. The nonlinearity lets it learn curved boundaries between concepts that a flat projection can't capture. This is what most modern VLMs actually use.
Dynamic Resolution
Real images come in all sizes and aspect ratios. AnyRes tiles a high-res image into patches processed separately, then merges. NaFlex uses flexible positional encodings. Both avoid distortion from resizing.
Comparing Architectures
A decision framework: Linear/MLP preserves all detail but costs more LLM tokens. Q-Former/Perceiver compresses aggressively but may lose fine-grained info. The right choice depends on your task — OCR needs full tokens, classification doesn't.
Sign up to unlock this chapter
This chapter is part of PixelBank Premium. Create a free account, then upgrade to read the full lesson — concepts, walkthroughs, and exercises.