Loading...
Master the bridge architectures that connect vision encoders to language models. From simple linear projections to sophisticated Q-Former and Perceiver Resampler designs, understand how visual representations are translated into the language model's embedding space and how different approaches trade off between token efficiency, information preservation, and computational cost.
The alignment problem in VLMs is fundamentally about bridging two representation spaces that were trained independently: the vision encoder's feature space (optimized for visual similarity via contrastive learning) and the LLM's embedding space (optimized for next-token prediction of text). These spaces encode different types of information in different geometric structures, and the bridge module must translate between them faithfully.
The bridge architecture is arguably the most critical design decision in a VLM. It determines how much visual information reaches the language model, how many tokens the LLM must process (directly impacting inference cost), and whether the model can preserve fine-grained spatial details or must compress to a coarse summary.
The field has explored a rich design space:
This chapter covers each approach in detail, with mathematical formulations, implementation walkthroughs, and a systematic comparison of trade-offs.
Click any topic to jump in
Vision encoders and LLMs speak different languages. CLIP groups by visual similarity (dogs and cats are close), LLMs group by linguistic meaning (dogs and cats are far apart). A bridge module must translate between these two worlds.
Each takes a different approach to the same problem
The simplest possible bridge: one matrix multiply converts each image patch independently. Surprisingly competitive — if CLIP and the LLM are both strong, a simple coordinate transform is enough.
32 learnable "question" tokens cross-attend to the image, each specializing to extract different info (objects, colors, layout). Compresses 257 image tokens down to just 32 — an 8x reduction.
Fixed latent tokens iteratively refine themselves by repeatedly cross-attending to image features. Similar goal to Q-Former (compression) but with a cleaner, more principled architecture.
A 2-layer MLP (linear → GELU → linear) replaces the single linear projection. The nonlinearity lets it learn curved boundaries between concepts that a flat projection can't capture. This is what most modern VLMs actually use.
Real images come in all sizes and aspect ratios. AnyRes tiles a high-res image into patches processed separately, then merges. NaFlex uses flexible positional encodings. Both avoid distortion from resizing.
A decision framework: Linear/MLP preserves all detail but costs more LLM tokens. Q-Former/Perceiver compresses aggressively but may lose fine-grained info. The right choice depends on your task — OCR needs full tokens, classification doesn't.
This chapter is part of PixelBank Premium. Create a free account, then upgrade to read the full lesson — concepts, walkthroughs, and exercises.