Vision-Language2023

BLIP-2

Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi

Paper Overview

BLIP-2 (Li, Li, Savarese & Hoi — Salesforce Research, ICML 2023) asks what looks like an accounting question and turns it into an architecture. End-to-end vision-language pre-training had become ruinously expensive, and the field's response was mostly to spend more. BLIP-2's response is to spend almost nothing: take an off-the-shelf image encoder, take an off-the-shelf LLM, freeze both, and train a small module in between.

The difficulty is that a frozen image encoder and a frozen LLM have never met. The LLM has never seen a pixel; the vision encoder has never been asked to produce something a language model can read. Earlier work tried to close this gap with an image-to-text generation loss alone, which BLIP-2 argues is insufficient — catastrophic forgetting is avoided by freezing, but alignment then has nowhere to come from.

The answer is the Querying Transformer (Q-Former): 32 learned query vectors that interrogate the frozen visual features and emit a fixed-size, language-shaped summary. It is trained in two stages — first to make those queries extract text-relevant visual information, then to make their output legible to a frozen LLM as a soft prompt.

The payoff is stark. BLIP-2 trains 188M parameters and scores 65.0 on zero-shot VQAv2; Flamingo80B trains 10.2B and scores 56.3. Same task, 54x fewer trainable parameters, 8.7 points better.

Chapter Roadmap

Click any topic to jump in

The Modality Gap

Freeze a vision encoder and an LLM and you save enormous compute — but neither has ever seen the other's representation space.

bridged by

Q-Former

188M trainable parameters and 32 learned queries form an information bottleneck between a frozen ViT and a frozen LLM.

first trained via

Stage 1 — Representation Learning

Three objectives (ITC, ITG, ITM) share one model and differ only in the self-attention mask between queries and text.

then trained via

Stage 2 — Generative Learning

A single fully-connected layer projects the 32 query outputs into the LLM's embedding space as soft visual prompts.

scaled with

Pre-training Recipe

129M images, CapFilt synthetic captions, and interchangeable frozen backbones — CLIP ViT-L/14, EVA-CLIP ViT-g/14, OPT, FlanT5.

produces

Outcomes and caveats

Results

65.0 on zero-shot VQAv2 with 188M trainable parameters — beating Flamingo80B's 56.3 while training 54x fewer.

Limitations

No in-context learning, inherited LLM failure modes, and a frozen-model design that cannot repair either.

Vision-language pre-training got good by getting big, and getting big got expensive. BLIP-2 starts from the observation that the expensive part — learning to see, and learning to speak — has already been paid for by somebody else. Every new multimodal model was re-learning both from scratch, at a scale only a handful of labs could afford.

So freeze them. Keeping both unimodal models frozen cuts the compute cost and sidesteps catastrophic forgetting — a model that never updates cannot degrade. But freezing immediately creates the problem the rest of the paper exists to solve: the LLM has never seen an image during its unimodal pre-training, and the vision encoder was never asked to emit something a language model could read. The ViT produces features tuned for visual tasks; the LLM expects token embeddings from its own learned distribution. There is no shared coordinate system, and — because both are frozen — neither side is allowed to move toward the other.

The obvious patch is to train the connector with an image-to-text generation loss and hope alignment falls out. This is what Frozen and Flamingo do, and BLIP-2's central claim is that it is insufficient: the signal is weak and indirect, and the paper's own ablation backs this up — removing the stage-1 representation-learning objectives causes a substantial drop.

BLIP-2's answer is a lightweight Querying Transformer pre-trained in two stages. Stage 1 attaches it to the frozen image encoder and forces its queries to extract visual information that is maximally informative about text. Stage 2 attaches the result to a frozen LLM and teaches those queries to speak in soft prompts. Each stage bootstraps from what is already there — hence the name.

It helps to see it as compression against a language-shaped codebook. The frozen ViT emits $257 \times 1024$ features; the LLM wants a short prefix of its own embeddings. The Q-Former must learn a map into a $32 \times 768$ space that discards everything the language model will not use. The bottleneck is not a limitation here — it is the mechanism.

Key Points

Pre-trained vision models offer high-quality visual representations; LLMs offer strong generation and zero-shot transfer — training either from scratch for a multimodal model wastes that investment

Keeping both frozen cuts compute cost and sidesteps catastrophic forgetting — the unimodal models cannot degrade if they never update

But freezing creates the real problem: the LLM has never seen images during its unimodal pre-training, so nothing in it is prepared to read visual features

Prior frozen-model methods (Frozen, Flamingo) rely on an image-to-text generation loss, which BLIP-2 argues is insufficient to bridge the gap on its own

The bridge must therefore be a trainable module that is small (cheap), and whose output is already shaped like something an LLM can consume

Two stages, because the queries must first learn what to extract before they can learn how to say it

Premium Content

Upgrade to PixelBank Premium to unlock this content.

Back to All Papers

Study Plans

VLM Study Plan LLM Study Plan All Papers Practice Problems

BLIP-2

Paper Overview

Chapter Roadmap

The Modality Gap

Q-Former

Stage 1 — Representation Learning

Stage 2 — Generative Learning

Pre-training Recipe

Results

Limitations

The Modality Gap

Key Points

Further Reading

Q-Former — The Querying Transformer

Stage 1 — Vision-Language Representation Learning

Stage 2 — Vision-to-Language Generative Learning

Pre-training Recipe

Results

Limitations

Premium Content

Related Papers

Study Plans