Chapter 4: Vision-Language Alignment

A linear projection $h = Wf + b$ performs an affine transformation that can rotate, scale, and translate the vision features into LLM space. Geometrically, it maps the vision manifold to a linear subspace of the LLM embedding space. The key limitation: if two concepts (e.g., Golden Retriever vs Labrador) are linearly inseparable in vision space, no linear $W$ can separate them in the output. This is why MLP projectors with nonlinear activations outperform linear on fine-grained tasks.

Stage 1 solves a constrained optimization $\min_W \|W F_\text{vision} - E_\text{LLM}\|^2$ where $E_\text{LLM}$ is fixed. With frozen endpoints, this is a least-squares problem — convex, with a closed-form solution $W^* = E_\text{LLM} F_\text{vision}^\dagger$ (the pseudoinverse). Unfreezing the LLM in Stage 2 makes the optimization non-convex because both $W$ and the LLM's embedding space $E$ now co-adapt, but the Stage 1 solution provides a warm start near a good basin. Without this warm start, jointly optimizing $W$ and $\theta_\text{LLM}$ from scratch risks converging to a poor local minimum where the LLM ignores the visual tokens entirely.

The Q-Former queries form a learned basis $Q \in \mathbb{R}^{32 \times d}$ that spans a 32-dimensional subspace of the image feature space. Cross-attention computes $Z = \text{softmax}(QW_Q(FW_K)^T / \sqrt{d_k}) \cdot FW_V$, which is a soft selection: each query computes a weighted average over all image patches, with weights determined by learned query-key compatibility. The 32 outputs are an information-theoretically compressed representation — they retain features with high mutual information with the text distribution while discarding low-relevance spatial detail.

The attention matrix $A \in \mathbb{R}^{32 \times 257}$ has rows that each sum to 1 via softmax, so each row defines a probability distribution over image patches — the Q-Former learns 32 different probabilistic "views" of the image. The rank of $A$ determines the effective number of independent queries: if $\text{rank}(A) < 32$, some queries extract linearly dependent information and are redundant. In practice, the self-attention layers between cross-attention steps encourage the queries to diversify, pushing $\text{rank}(A)$ toward 32 by allowing queries to coordinate and avoid redundant attention patterns.

Each objective imposes a different attention mask structure on the same transformer weights. ITC uses a block-diagonal mask (queries and text attend only within their modality — no cross-modal flow). ITM uses a full bidirectional mask (every token attends to every other token). ITG uses a causal mask (each text token attends only to previous text tokens and all queries). This progression — independent $\to$ bidirectional $\to$ causal — systematically increases the model's capacity to mix visual and textual information, training the shared weights to handle all three regimes of cross-modal interaction.

The Perceiver iteratively refines latent tokens $L^{(i+1)} = \text{CrossAttn}(L^{(i)}, F) + L^{(i)}$ with residual connections. Each layer can be viewed as a gradient step: the cross-attention pulls $L$ toward image features $F$ that have high attention weight, while the residual connection prevents catastrophic forgetting of earlier refinements. After $N$ layers, the latent tokens converge to a fixed point that balances information from all layers of cross-attention.

Learned initialization $L^{(0)}$ can be viewed as an inductive bias that places the latent tokens near a good basin of attraction in feature space before cross-attention begins. With random init $z^{(0)} \sim \mathcal{N}(0, \sigma^2 I)$, the first several layers must "search" for the relevant subspace — the attention weights start nearly uniform across patches because the queries have no preference. With learned init, the first layer's attention is already structured, giving a convergence advantage of roughly $O(1)$ vs $O(1/N)$ in effective information extraction per layer, where $N$ is the Perceiver depth.

Attention pooling computes a single weighted average $z = \sum_i \alpha_i f_i$, which lies in the convex hull of the input features — a single point, regardless of the input's dimensionality. The Perceiver's $K$ latent tokens each attend differently, so the output spans up to $K$ linearly independent directions in feature space. With $K$ latents, the output can represent a $K$-dimensional subspace of the input, while pooling collapses everything to a 1D projection. This is precisely why pooling loses spatial information: it projects a high-rank feature set onto a rank-1 output.

The MLP $h = W_2 \cdot \text{GELU}(W_1 f + b_1) + b_2$ introduces a nonlinear feature transformation. The hidden layer $\text{GELU}(W_1 f + b_1)$ creates nonlinear decision boundaries in the vision feature space, enabling the projection to separate features that are linearly entangled. With hidden dimension $d_h > d_\text{vision}$, the first layer "lifts" features to a higher-dimensional space where linearly inseparable clusters become separable (Cover's theorem), then $W_2$ projects back to $d_\text{LLM}$.

With $\alpha$ initialized to 0, $\tanh(0) = 0$, so the adapter output is exactly zero at initialization — the LLM behaves identically to its pre-trained state. This is a strict identity initialization: $y_l = x_l + 0 \cdot \text{CrossAttn}(\cdot) = x_l$. During training, $\alpha$ grows and $\tanh(\alpha)$ smoothly opens the gate. Since $\frac{d}{d\alpha}\tanh(\alpha)\big|_{\alpha=0} = 1$, the gradient flows cleanly through the gate in early training — no vanishing gradient problem. This is equivalent to a residual connection with a learned step size, where the step size is parameterized to start at zero and grow monotonically.

The design space can be characterized by three quantities: depth $d$ (number of layers), width ratio $w = d_\text{hidden}/d_\text{vision}$, and compression ratio $c = N_\text{out}/N_\text{in}$. Empirically, accuracy scales as $\text{Acc} \propto \log(d \cdot w) - \lambda c$ — deeper and wider projectors help logarithmically, but token compression hurts linearly. The 2-layer MLP with $w \approx 1$ and $c = 1$ (no compression) sits at the Pareto optimum: adding a third layer gives diminishing returns ($<0.3\%$ accuracy), while any compression ($c < 1$) costs more than the compute it saves.

For an input image of size $H \times W$, AnyRes computes a grid of $\lceil H/s \rceil \times \lceil W/s \rceil$ tiles where $s$ is the ViT's native resolution (e.g., 336). Each tile produces $N_p$ tokens, giving $N_\text{total} = n_\text{tiles} \times N_p + N_p$ (tiles + downscaled global view). The total token count grows linearly with image area: a 672×672 image produces $4 \times 576 + 576 = 2{,}880$ tokens. This trades LLM compute (quadratic in sequence length) for spatial resolution.

Standard ViTs require fixed resolution because positional embeddings $P \in \mathbb{R}^{N \times d}$ are learned for exactly $N = (H/p)^2$ patches. NaFlex decouples position from resolution by using 2D interpolation: for input size $H' \times W'$, it interpolates $P$ to $P' \in \mathbb{R}^{(H'/p)(W'/p) \times d}$. This preserves the relative spatial structure while adapting to arbitrary aspect ratios without tiling overhead — a single forward pass handles any resolution.

Average pooling with a $2 \times 2$ kernel computes $z_{ij} = \frac{1}{4}\sum_{k \in \mathcal{N}(i,j)} f_k$, which is a low-pass filter in the spatial frequency domain — it preserves DC and low-frequency components while attenuating high-frequency spatial detail. The information loss is $I(F; T) - I(\text{Pool}(F); T)$, which is small when the downstream task depends on global semantics (scene understanding, captioning) but large when it requires high-frequency spatial precision (character recognition, small object counting). This is why pooled models degrade on OCR benchmarks but retain nearly full accuracy on VQA.

The trade-off can be formalized as $\text{Score}(n) = \alpha \cdot \text{Accuracy}(n) - \beta \cdot \text{Cost}(n)$ where $n$ is the token count. Accuracy is concave in $n$ (diminishing returns — going from 32→64 tokens helps more than 512→1024), while Cost is superlinear (LLM attention is $O(n^2)$). The Pareto-optimal bridge depends on the curvature of both functions for your specific task: OCR has steep accuracy gains from more tokens; classification has a flat curve beyond ~32 tokens.

The decision reduces to $\arg\min_b \text{Cost}(b)$ subject to $\text{Accuracy}(b) \geq \tau$ for a task-dependent threshold $\tau$. LLM attention cost scales as $O(n^2)$ where $n$ is total sequence length, so Q-Former's 32 tokens vs MLP's 576 tokens gives a $(576/32)^2 = 324\times$ reduction in the visual-token attention block. At high $\tau$ (demanding applications like medical VQA), only MLP-2 satisfies the accuracy constraint. At low $\tau$ (chatbot, mobile), Q-Former dominates on cost. The crossover point is task-dependent — for VQAv2, $\tau \approx 75\%$ is roughly where Q-Former becomes viable.

The accuracy-per-token metric $\eta = \text{Acc}/N_\text{tokens}$ reveals that Q-Former achieves $\eta = 2.3$ ($74.2\%/32$) vs MLP's $\eta = 0.14$ ($80.0\%/576$) — Q-Former extracts 16x more accuracy per token. But this metric is misleading: the marginal value of additional tokens follows $d\text{Acc}/dN \propto N^{-\gamma}$ with $\gamma \approx 0.5$, meaning the first 32 tokens carry most of the information (a Zipf-like distribution over visual tokens by importance). The remaining 544 tokens contribute only $+5.8\%$ absolute accuracy, confirming that visual information is highly concentrated in a few principal components.