Chapter 3: Language Model Backbone

Full self-attention computes $A = \text{softmax}(QK^T / \sqrt{d_k})$ with no masking, yielding a doubly-stochastic-like matrix where information flows in all directions. Each output $h_i = \sum_j A_{ij} V_j$ is a convex combination of all value vectors, giving the encoder a global receptive field in a single layer — unlike the causal case where layer $\ell$ has receptive field $\ell$.

Cross-attention factorizes as $\text{softmax}(Q_{\text{dec}} K_{\text{enc}}^T / \sqrt{d_k}) V_{\text{enc}}$, where queries come from a different distribution than keys. This computes a soft retrieval: for each decoder token, the dot product $q_i^T k_j$ measures relevance, and softmax converts scores into a probability distribution over encoder positions. The output is the expected value under this retrieval distribution.

The encoder output $\text{Enc}(x_{1:N})$ is computed once with $O(N^2 d)$ attention cost, then reused at every decoder step. Generating $T$ tokens costs $O(T \cdot N \cdot d)$ for cross-attention (linear in $N$ per step), compared to $O(T \cdot (N+T)^2 \cdot d)$ for a decoder-only model reprocessing the full prefix. The amortized encoder cost makes this architecture more efficient when $T \gg 1$.

The bridge projection $v_i = W f_i^{\text{vision}} + b$ is a learned affine map $\mathbb{R}^{d_{\text{vis}}} \to \mathbb{R}^{d_{\text{LLM}}}$ that aligns the visual feature manifold with the LLM's embedding space. After projection, the Frobenius norm $\|V - T\|_F$ between visual and text token matrices decreases as training progresses, indicating the two modalities converge to a shared representation.

Delimiter embeddings $e_{[\text{IMG}]}, e_{[\text{/IMG}]} \in \mathbb{R}^{d_{\text{LLM}}}$ are learnable vectors that partition the input sequence into modality-specific segments. The attention pattern naturally develops block structure: tokens within the $[\text{IMG}] \ldots [\text{/IMG}]$ segment attend strongly to each other, creating an implicit modality indicator without explicit type embeddings.

With $k$ images of $N_i$ tokens each and $T$ text tokens, the total sequence length is $L = \sum_{i=1}^k (N_i + 2) + T$ (including delimiters). Self-attention cost scales as $O(L^2 d)$, so doubling the number of images approximately quadruples the attention cost. This quadratic scaling motivates token compression methods that reduce each $N_i$ while preserving information.

For a mixed sequence of $N$ image and $T$ text tokens, the causal attention matrix has $\frac{(N+T)(N+T+1)}{2}$ non-zero entries. Text token $t_j$ attends to all $N$ image tokens plus $j$ preceding text tokens, so its attention entropy is $H_j \leq \log(N+j)$. As $N$ grows large relative to $T$, image tokens dominate each text token's attention budget.

The gated residual $y = x + \tanh(\alpha) \cdot \text{CrossAttn}(x, v)$ preserves the Jacobian $\partial y / \partial x = I$ at initialization (when $\alpha=0$), ensuring stable gradient flow through the pretrained LLM layers. As $\alpha$ is learned, $\tanh(\alpha) \in (-1, 1)$ acts as a smooth interpolation coefficient, and $\frac{d}{d\alpha}\tanh(\alpha) = 1 - \tanh^2(\alpha)$ peaks at $\alpha=0$, so the gate is most sensitive to gradient updates early in training.

Attention sinks arise because softmax normalization $a_j = e^{s_j} / \sum_k e^{s_k}$ forces all weights to be positive and sum to 1. Early tokens accumulate high attention not because they are semantically important, but because the model uses them as a "no-op" sink when no token is strongly relevant — the residual stream's skip connection ensures useful information still propagates even when attention is diffuse.

RoPE encodes position by rotating query-key pairs: $(q_i, k_j) \mapsto (R_i q_i, R_j k_j)$ where $R_\theta$ is a block-diagonal rotation matrix. The inner product $q_i^T R_i^T R_j k_j = q_i^T R_{j-i} k_j$ depends only on relative position $j-i$, providing translation invariance. GQA reduces the KV cache from $O(n_h \cdot d_h \cdot L)$ to $O(n_{\text{kv}} \cdot d_h \cdot L)$ with $n_{\text{kv}} \ll n_h$ KV heads.

Sliding Window Attention restricts each token to attend within a window of size $w$, reducing per-layer attention cost from $O(L^2)$ to $O(Lw)$. Over $\ell$ layers, the effective receptive field expands to $\ell \cdot w$ tokens. Mixtral's top-$k$ routing selects $k=2$ experts from $E=8$, so active parameters per token are $\frac{k}{E} = 25\%$ of total, achieving the quality of a dense 12.9B model with the memory footprint of 46.7B.

Data quality scaling suggests an alternative to Chinchilla: $\text{Performance} \approx f(N, D_{\text{eff}})$ where $D_{\text{eff}}$ is the effective dataset size after quality filtering. Phi demonstrates that aggressive data curation ($D_{\text{eff}} \ll D_{\text{raw}}$) can shift the Pareto frontier, achieving $L(N_{\text{small}}, D_{\text{curated}}) \approx L(N_{\text{large}}, D_{\text{noisy}})$ at a fraction of the inference cost.

Vocabulary size $|V|$ determines the embedding table size $|V| \times d$ and the output projection cost $O(|V| \cdot d)$ per token. CJK languages require ~3.5x more tokens under a 32K BPE vocabulary than under 152K, because characters are split into byte-level fallback tokens. A larger vocabulary reduces sequence length (and hence attention cost) by a factor proportional to the compression ratio gain.

The empirical loss follows $L(N, D) = \frac{A}{N^\alpha} + \frac{B}{D^\beta} + L_\infty$ with $\alpha \approx 0.34, \beta \approx 0.28$. Minimizing $L$ subject to the compute constraint $C \approx 6ND$ yields the optimal ratio $D^*/N^* \approx 20$. Over-training (using $D \gg 20N$) sacrifices training compute efficiency for better per-parameter quality, which is optimal when inference cost dominates the total deployment budget.

Total VLM FLOPs decompose as $C_{\text{total}} = C_{\text{encoder}} + C_{\text{LLM}}$ where $C_{\text{LLM}} \approx 2N(N_v + N_t)$ per token generated. The attention cost within the LLM scales as $O((N_v + N_t)^2)$, creating a quadratic dependence on visual token count. The vision encoder adds a fixed $O(N_v^2 \cdot d_{\text{vis}})$ cost per image, making the total cost a sum of two quadratics in $N_v$.

Quantization maps floating-point weights to a discrete grid: $w_q = \text{round}(w / s) \cdot s$ where $s$ is the scale factor. The quantization error $\|W - W_q\|_F$ grows as bit-width decreases, but outlier-aware methods (AWQ) weight the error by activation magnitude: $\min_{s} \|W \cdot \text{diag}(a) - W_q \cdot \text{diag}(a)\|_F$, preserving the channels that matter most for the output distribution.