BLIP-2

Vision-language pre-training got good by getting big, and getting big got expensive. BLIP-2 starts from the observation that the expensive part — learning to see, and learning to speak — has already been paid for by somebody else. Every new multimodal model was re-learning both from scratch, at a scale only a handful of labs could afford.

So freeze them. Keeping both unimodal models frozen cuts the compute cost and sidesteps catastrophic forgetting — a model that never updates cannot degrade. But freezing immediately creates the problem the rest of the paper exists to solve: the LLM has never seen an image during its unimodal pre-training, and the vision encoder was never asked to emit something a language model could read. The ViT produces features tuned for visual tasks; the LLM expects token embeddings from its own learned distribution. There is no shared coordinate system, and — because both are frozen — neither side is allowed to move toward the other.

The obvious patch is to train the connector with an image-to-text generation loss and hope alignment falls out. This is what Frozen and Flamingo do, and BLIP-2's central claim is that it is insufficient: the signal is weak and indirect, and the paper's own ablation backs this up — removing the stage-1 representation-learning objectives causes a substantial drop.

BLIP-2's answer is a lightweight Querying Transformer pre-trained in two stages. Stage 1 attaches it to the frozen image encoder and forces its queries to extract visual information that is maximally informative about text. Stage 2 attaches the result to a frozen LLM and teaches those queries to speak in soft prompts. Each stage bootstraps from what is already there — hence the name.

It helps to see it as compression against a language-shaped codebook. The frozen ViT emits $257 \times 1024$ features; the LLM wants a short prefix of its own embeddings. The Q-Former must learn a map into a $32 \times 768$ space that discards everything the language model will not use. The bottleneck is not a limitation here — it is the mechanism.