Chapter 4: Pretraining

At each position $t$ the model outputs a probability distribution over the vocabulary, given the earlier tokens $x_{<t}$ and weights $\theta$. The loss for that position is $-\log P(x_t \mid x_{<t}; \theta)$, the negative log of the probability given to the token that actually came next. Averaging over all $T$ positions gives the formula. A confident correct prediction costs almost nothing, and a confident wrong one costs a lot, because $-\log p$ grows without bound as $p$ goes to 0. Loss is measured in nats with the natural log. Its value depends on the tokenizer, so losses from different vocabularies are not directly comparable.

Perplexity is the exponential of the average cross-entropy, as in the formula. If a model's loss is $\mathcal{L}$ nats, its perplexity is $e^{\mathcal{L}}$. It reads as an effective branching factor: a perplexity of 10 means the model is as uncertain, on average, as if it were choosing uniformly among 10 tokens. Lower is better, and a perfect model would reach 1. Because it exponentiates the loss, small loss changes become visible ratios. GPT-3 reached a zero-shot perplexity of about 20 on Penn Treebank. Perplexity depends on the tokenizer and dataset, so compare it only between models that share both.

During training, the input at every position is the true text, not the model's own earlier predictions. This is teacher forcing. Because every position's input is known in advance, one forward pass with a causal mask computes the loss at all $T$ positions in parallel, which is what makes pretraining on trillions of tokens feasible. The cost is a mismatch with generation, where the model feeds on its own samples. It never practised recovering from its own mistakes, so one bad token can push it into text unlike anything it saw. This is called exposure bias, and sampling strategies partly compensate for it.

A token-level objective scores each next-token prediction separately against the known text, so every position gives a gradient and the signal is dense and low-variance. A sequence-level objective scores a whole generated output, for example with a reward model, a human preference, or a test that runs generated code. That matches what users care about, but the model must first generate text, the score is a single number for many tokens, and the gradient must be estimated with methods such as REINFORCE or PPO. So the standard recipe is token-level pretraining followed by sequence-level fine-tuning with RLHF.

Causal language modeling uses the chain rule of probability, as in the formula: the probability of a sequence is the product of each token's probability given all earlier tokens. The model predicts token $t$ from $x_1$ to $x_{t-1}$, and a causal mask stops it from seeing later positions. Every position except the first is a training target, so almost all the text gives signal. The same factorisation gives generation: sample a token, append it, and repeat. GPT, Llama, Mistral, and PaLM are all causal models. The price is that each token's representation sees only its left context.

Masked language modeling, introduced with BERT, picks 15 percent of positions at random, the set $\mathcal{M}$, and asks the model to recover them from everything else, $x_{\backslash \mathcal{M}}$, as in the formula. There is no causal mask, so each prediction uses context on both sides. Of the chosen positions, 80 percent become a [MASK] token, 10 percent a random token, and 10 percent stay unchanged, so the model cannot rely on seeing [MASK], which never appears in real use. The representations are strong for classification and extraction. But only the masked positions give a loss, and the model cannot generate text naturally.

Prefix language modeling splits a sequence into a prefix, $x_1$ to $x_p$, and a continuation. Inside the prefix the attention is bidirectional, so every prefix token sees the whole prefix. The continuation is generated causally, and the loss covers only continuation tokens, which gives the conditional probability in the formula. That suits tasks with a clear input and output, such as translation or question answering. UL2 and the prefix-LM variants in T5's study use it. Encoder-decoder models such as T5 get a similar effect with separate stacks. Wang et al. (2022) found the best choice depends on whether multitask fine-tuning follows.

Masked models led language understanding benchmarks in 2018 and 2019, yet today's large models are almost all causal decoders. Four reasons stand out. Generation is the main use of an LLM, and a causal model generates naturally. Every token is a training target instead of 15 percent. One decoder stack with one objective is simple to scale and to serve with a key-value cache. And at scale, causal models learn tasks from prompts through in-context learning, which removes the need for a task-specific head. Wang et al. (2022) found causal decoders the best zero-shot models after pretraining alone.

Most pretraining corpora start from Common Crawl, a public archive of monthly web crawls, and add curated sources. The Pile (Gao et al., 2020) is an 825 GiB English corpus built from 22 subsets, including PubMed, arXiv, GitHub, and books. RefinedWeb (Penedo et al., 2023) showed that carefully filtered web data alone can match curated mixtures, and it extracted five trillion tokens from Common Crawl. RedPajama reproduced the LLaMA recipe in the open, and FineWeb documented its filtering choices. Proprietary models add licensed and synthetic data. A dataset's token count depends on the tokenizer, so cross-paper figures are approximate.

Raw crawl data goes through a sequence of filters, each removing a large fraction. First, text extraction strips HTML, menus, and boilerplate. Language identification then keeps the target languages. Heuristic quality rules drop pages with too few words, too many symbols, or repeated lines, and a classifier may score pages by how much they resemble reference text such as Wikipedia. Deduplication removes copies. PII scrubbing and toxicity filters come last. Each filter trades quantity for quality, and aggressive quality classifiers can remove dialects and minority topics along with the spam.

Web text is full of repeats: syndicated news, licence text, templates, and scraped copies. Exact deduplication hashes documents or long substrings. Near-duplicate detection uses MinHash, which estimates the Jaccard similarity of two documents' sets of word n-grams, so pages differing by a few words still match. Lee et al. (2021) found that more than 1 percent of an LM's unprompted output was copied verbatim from training data, and one 61-word sentence appeared over 60,000 times in C4. Deduplicated models emitted memorised text ten times less often and needed fewer steps for the same accuracy. Too loose a similarity threshold can merge genuinely different documents.

Training draws from several sources at chosen sampling weights, and those weights shape the skills. The LLaMA mix was 67 percent Common Crawl, 15 percent C4, 4.5 percent each of GitHub, Wikipedia, and books, 2.5 percent arXiv, and 2 percent StackExchange. High-quality sources are often upsampled and seen more than once per run. More code tends to help code generation and structured reasoning, and more math helps arithmetic, but every percent given to one source is taken from another. Some runs use a curriculum, shifting toward cleaner or more specialised data near the end of training. Good weights are usually found with small proxy models.

In data parallelism every GPU holds a full copy of the model and processes a different slice of the batch, called its micro-batch. After the backward pass, an all-reduce averages the gradients across GPUs, so every copy applies the same update and the copies stay identical. Gradient accumulation adds several micro-batches' gradients before each update, so the effective batch in the formula can exceed memory limits. Throughput scales almost linearly while communication keeps up. The limit is memory, because each GPU still stores the whole model and optimizer state. ZeRO addresses this by sharding those states across the data-parallel GPUs.

When the model does not fit on one GPU, it is split. Tensor parallelism (Megatron-LM) splits individual matrices. The first feed-forward matrix is split by columns and the second by rows, so each GPU computes part of the layer and one all-reduce combines the results. That needs fast links, so it usually stays inside one 8-GPU node. Pipeline parallelism assigns groups of layers to different GPUs and passes activations forward. Micro-batches keep the stages busy, but GPUs sit idle while the pipeline fills and drains. Large runs combine tensor, pipeline, and data parallelism.

Mixed precision runs the forward and backward passes in a 16-bit format and keeps an FP32 master copy of the weights for the update, as in the formula. Sixteen-bit matrix multiplies run several times faster on tensor cores, and 16-bit activations halve activation memory. The master copy matters because tiny updates, about the learning rate times the gradient, would round to nothing in 16 bits. BF16 keeps FP32's 8 exponent bits and gives up mantissa precision, so it has FP32's range and rarely overflows. FP16 has more precision but a narrow range, so it needs loss scaling to stop small gradients underflowing to zero.

AdamW is the default optimizer for LLM pretraining. It keeps two running averages per parameter, as in the formula: the momentum $m_t$ of the gradients $g_t$ and the average of their squares $v_t$. It updates each weight by $\eta \, m_t / (\sqrt{v_t} + \epsilon)$, giving each parameter its own step size, and it applies weight decay directly to the weights. The two states cost 8 bytes per parameter in FP32. Adafactor stores factored second moments, Lion keeps only momentum, and 8-bit Adam quantises both states. The schedule is usually a linear warmup followed by cosine decay.

Training compute is approximately $C \approx 6ND$ FLOPs, as in the formula, where $N$ is the parameter count and $D$ the number of training tokens. The forward pass costs about $2N$ FLOPs per token, one multiply and one add per weight. The backward pass costs about twice that, $4N$, because it computes gradients with respect to both activations and weights. That gives 6 FLOPs per parameter per token. The rule ignores attention's sequence-length term, which is small next to $N$ for typical context lengths. Real clusters sustain 30 to 50 percent of peak FLOPS, so divide by utilisation when converting to time.

Kaplan et al. (2020) found that test loss falls as a power law in compute across several orders of magnitude, as in the formula. $C_0$ is a fitted constant, and the exponent $\alpha_C$ is about 0.05, so each 10 times more compute multiplies the reducible part of the loss by $10^{-0.05} \approx 0.89$. $L_\infty$ is the irreducible loss, the entropy of the text that no model can remove. Teams fit these curves on runs costing about a thousandth of the target and extrapolate to choose the final configuration. Downstream abilities are less predictable, since some benchmarks jump suddenly as loss falls smoothly.

For a fixed compute budget $C$, a larger model sees fewer tokens. Kaplan et al. (2020) advised putting most extra compute into parameters, and GPT-3 used 175B parameters with only 300B tokens, about 1.7 tokens per parameter. Hoffmann et al. (2022) trained over 400 models and found that parameters and tokens should grow together, each roughly as $C^{0.5}$, which works out to about 20 tokens per parameter. Their 70B Chinchilla, trained on 1.4T tokens, beat the 280B Gopher at the same compute. Epoch AI's replication later refitted the loss formula but supported roughly the same optimal ratio.

Chinchilla minimises training loss for a given training budget, but a deployed model keeps costing compute on every token it serves, about $2N$ FLOPs per token. If a model will serve trillions of tokens, a smaller model trained on more data can be cheaper overall, even though it reaches a given loss less efficiently during training. LLaMA made this choice explicitly, training a 7B model on 1T tokens. Llama 2 7B saw 2T tokens, about 286 per parameter, and Llama 3 8B saw 15T, about 1,875. Over-training has diminishing returns, but it has not hit a hard wall yet.