Loading...
Master the complete training pipeline that transforms a randomly initialized model into a capable, aligned visual assistant. From large-scale pretraining on billions of image-text pairs through supervised fine-tuning on curated instructions to reinforcement learning from human feedback, understand each stage's objectives, data requirements, and failure modes — including the persistent challenge of hallucination.
Training a modern VLM is not a single optimization run — it is a carefully orchestrated sequence of training stages, each with distinct objectives, data distributions, and hyperparameter regimes. Getting this recipe wrong at any stage cascades into downstream failures that are difficult to diagnose.
The standard VLM training pipeline has evolved into three (sometimes four) distinct phases:
Pretraining on massive, noisy web data teaches the model basic visual-language associations. The objective is broad coverage, not precision. Models process billions of image-text pairs from datasets like LAION-5B and DataComp, learning that "dog" co-occurs with images of dogs, that spatial prepositions have visual correlates, and that captions describe salient image content.
Supervised Fine-Tuning (SFT) on carefully curated instruction data transforms the pretrained model into an instruction-following assistant. Here, quality dominates quantity: a few hundred thousand high-quality examples outperform millions of noisy ones. The critical insight is loss masking — computing gradients only on the assistant's response tokens, not on the instruction or system prompt.
Alignment via RLHF or DPO adjusts the model's behavior to match human preferences. This stage addresses a fundamental gap: SFT teaches the model what to say in response to instructions, but not how to say it well — avoiding hallucination, being appropriately cautious, and calibrating confidence. RLHF uses a reward model trained on human preference comparisons, while DPO directly optimizes on preference pairs without a reward model.
The mathematical foundations of each stage differ significantly:
Each stage has its own failure modes: pretraining can memorize noise, SFT can overfit to formatting, and RLHF can reward-hack. Understanding these failure modes is as important as understanding the training objectives.
This chapter also addresses the persistent challenge of hallucination — arguably the single most important unsolved problem in VLMs. We examine why models fabricate objects, misattribute properties, and invent spatial relationships, and explore the mitigation strategies (RLHF, data quality, specialized objectives) that have shown promise.
Finally, we cover the training infrastructure required to run these recipes at scale: distributed training across hundreds of GPUs, mixed-precision arithmetic, gradient checkpointing, and the engineering challenges that make VLM training as much a systems problem as a research one.
Click any topic to jump in
ITC, ITM, masked LM, and next-token prediction — the four objectives that teach VLMs to connect vision and language.
Web-scale collection, deduplication, quality filtering, and safety — building billion-pair datasets like LAION-5B.
Loss masking, instruction curation, and chat templates — teaching the pretrained model to follow instructions.
RL-based vs direct optimization
Visual reward models and PPO — aligning VLM outputs with human preferences through reinforcement learning.
Direct preference optimization as a simpler alternative to RLHF — no reward model needed.
Quality problems and infrastructure needs
Why VLMs fabricate visual details, the CHAIR metric, and mitigation strategies from training to inference.
DeepSpeed, FSDP, mixed precision, gradient checkpointing — the systems needed for large-scale VLM training.
This chapter is part of PixelBank Premium. Create a free account, then upgrade to read the full lesson — concepts, walkthroughs, and exercises.