Chapter 1: Introduction to LLMs

The chain rule decomposes $P(x_1, \ldots, x_T) = \prod_t P(x_t \mid x_{<t})$, meaning any joint distribution over sequences can be modeled autoregressively. The softmax output $P(x_t \mid x_{<t}) = \text{softmax}(W h_t)$ maps a $d$-dimensional hidden state to a $V$-dimensional probability simplex. Training minimizes cross-entropy: $\mathcal{L} = -\sum_t \log P(x_t \mid x_{<t})$, which is equivalent to minimizing KL divergence from the true data distribution.

The parameter count $N$ typically scales as $N \approx 12 L d^2$ for a transformer with $L$ layers and dimension $d$. Doubling $d$ quadruples parameters. The compute budget follows $C \approx 6ND$ FLOPs where $D$ is training tokens. A 70B model trained on 2T tokens requires approximately $6 \times 70 \times 10^9 \times 2 \times 10^{12} = 8.4 \times 10^{23}$ FLOPs — thousands of GPU-years.

Emergence can be formalized as a phase transition in performance: below a threshold scale $N^*$, task accuracy $A(N) \approx \text{random}$, and above it $A(N)$ jumps sharply. Some researchers argue this is an artifact of nonlinear evaluation metrics — under log-linear metrics, performance often improves smoothly. The debate centers on whether $A(N)$ is truly discontinuous or just appears so under certain metrics.

Transfer learning exploits the factorization $P_{\text{task}}(y \mid x) = P_{\text{pretrained}}(y \mid x) \cdot \frac{P_{\text{task}}(y \mid x)}{P_{\text{pretrained}}(y \mid x)}$. Fine-tuning adjusts the ratio term with far fewer examples than learning $P_{\text{task}}$ from scratch. LoRA approximates the weight update as $\Delta W = BA$ where $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times d}$, with rank $r \ll d$, reducing trainable parameters from $d^2$ to $2dr$.