Chapter 9: Multimodal Reasoning & Applications

The previous chapters built VLMs that can perceive images and follow instructions. But perception alone is insufficient for many real-world tasks. A radiologist does not merely describe an X-ray -- they reason about anatomical relationships, compare with prior scans, and synthesize findings into a diagnosis. Similarly, the most impactful VLM applications require sophisticated reasoning over visual inputs.

This chapter explores seven dimensions of multimodal reasoning that transform VLMs from descriptive tools into intelligent visual agents:

1. Chain-of-thought reasoning: Structured step-by-step visual analysis that decomposes complex questions into observable facts, logical inferences, and conclusions
2. Visual grounding: Connecting language to specific image regions, enabling VLMs to point at what they describe
3. Document understanding: Extracting structured information from documents, receipts, charts, and academic papers without traditional OCR pipelines
4. Video understanding: Extending spatial reasoning to the temporal dimension, requiring frame selection strategies and motion comprehension
5. Multi-image reasoning: Comparing, contrasting, and synthesizing information across multiple images simultaneously
6. Agentic behavior: VLMs that can take actions -- navigating GUIs, calling tools, browsing the web, and executing multi-step plans grounded in visual feedback
7. Spatial reasoning: Understanding 3D structure, depth, spatial relationships, and geometric properties from 2D images

Each capability builds on the foundation of visual instruction tuning (Chapter 6) and architectural innovations (Chapters 7-8), pushing VLMs toward general-purpose visual intelligence.

The mathematical framework connecting these capabilities is the conditional generation paradigm:

$P(y_1, \ldots, y_T \mid x_{\text{img}}, x_{\text{text}}) = \prod_{t=1}^{T} P(y_t \mid y_{<t}, x_{\text{img}}, x_{\text{text}})$

What distinguishes these advanced applications is what the model must encode in $y$: spatial coordinates for grounding, structured JSON for documents, temporal narratives for video, comparative analysis for multi-image, and action sequences for agents. The same autoregressive framework accommodates all these outputs through careful prompt engineering and specialized training data.