Chapter 7: Training Recipes: Pretraining, SFT & RLHF

Training a modern VLM is not a single optimization run — it is a carefully orchestrated sequence of training stages, each with distinct objectives, data distributions, and hyperparameter regimes. Getting this recipe wrong at any stage cascades into downstream failures that are difficult to diagnose.

The standard VLM training pipeline has evolved into three (sometimes four) distinct phases:

1. Pretraining on massive, noisy web data teaches the model basic visual-language associations. The objective is broad coverage, not precision. Models process billions of image-text pairs from datasets like LAION-5B and DataComp, learning that "dog" co-occurs with images of dogs, that spatial prepositions have visual correlates, and that captions describe salient image content.

2. Supervised Fine-Tuning (SFT) on carefully curated instruction data transforms the pretrained model into an instruction-following assistant. Here, quality dominates quantity: a few hundred thousand high-quality examples outperform millions of noisy ones. The critical insight is loss masking — computing gradients only on the assistant's response tokens, not on the instruction or system prompt.

3. Alignment via RLHF or DPO adjusts the model's behavior to match human preferences. This stage addresses a fundamental gap: SFT teaches the model what to say in response to instructions, but not how to say it well — avoiding hallucination, being appropriately cautious, and calibrating confidence. RLHF uses a reward model trained on human preference comparisons, while DPO directly optimizes on preference pairs without a reward model.

The mathematical foundations of each stage differ significantly:

• Pretraining minimizes cross-entropy over massive token sequences: $\mathcal{L}_{\text{pretrain}} = -\sum_{t} \log p(x_t | x_{<t}, \mathbf{v})$
• SFT minimizes cross-entropy only on response tokens: $\mathcal{L}_{\text{SFT}} = -\sum_{t \in \text{response}} \log p(x_t | x_{<t}, \mathbf{v}, \text{instruction})$
• RLHF maximizes expected reward with KL penalty: $\max_{\pi} \mathbb{E}[r(x,y)] - \beta \cdot \text{KL}[\pi \| \pi_{\text{ref}}]$

Each stage has its own failure modes: pretraining can memorize noise, SFT can overfit to formatting, and RLHF can reward-hack. Understanding these failure modes is as important as understanding the training objectives.

This chapter also addresses the persistent challenge of hallucination — arguably the single most important unsolved problem in VLMs. We examine why models fabricate objects, misattribute properties, and invent spatial relationships, and explore the mitigation strategies (RLHF, data quality, specialized objectives) that have shown promise.

Finally, we cover the training infrastructure required to run these recipes at scale: distributed training across hundreds of GPUs, mixed-precision arithmetic, gradient checkpointing, and the engineering challenges that make VLM training as much a systems problem as a research one.