Loading...
Understand how LLMs are trained from scratch on massive text corpora. Learn about language modeling objectives (causal vs masked), the datasets and data pipelines that fuel pretraining, the distributed training infrastructure required, and the compute-optimal strategies that determine how to allocate your training budget.
Pretraining is the most expensive and foundational phase of building an LLM. During pretraining, the model learns language structure, factual knowledge, reasoning patterns, and coding ability by predicting tokens across trillions of training examples. The cost can range from thousands to hundreds of millions of dollars.
The key insight behind modern pretraining is remarkably simple: train a large transformer to predict the next token in a sequence, and let scale do the rest. This simple objective, applied to enough data with enough parameters, produces models capable of translation, summarization, code generation, mathematical reasoning, and more---none of which were explicitly trained for.
However, the simplicity of the objective belies the complexity of the engineering. Pretraining requires: carefully curated and deduplicated datasets, distributed training across thousands of GPUs, sophisticated optimization strategies, and careful monitoring for training instabilities. This chapter covers all these aspects.
This chapter covers:
Click any topic to jump in
Cross-entropy loss, perplexity, and teacher forcing — the training objectives that drive next-token prediction in every LLM.
Objectives determine architecture and data
Autoregressive vs bidirectional objectives — why causal modeling dominates for generation and how BERT's masking works.
Datasets, deduplication, and data mixtures — the fuel that determines what an LLM knows and how well it reasons.
Infrastructure and theory for training at scale
Data parallelism, model parallelism, and mixed precision — the distributed systems engineering behind pretraining.
Chinchilla-optimal compute allocation — predicting loss from parameters, data, and FLOPs before training begins.
The training objective defines what the model learns. For LLMs, the two primary objectives are next-token prediction (autoregressive/causal) and masked token prediction (bidirectional). The choice fundamentally determines what the model can do.
Key insight: The objective shapes the model's capabilities. Causal LM produces generators; masked LM produces understanding systems. Modern LLMs overwhelmingly use causal objectives.
The standard pretraining loss: the negative log-likelihood of the correct next token, averaged over all positions. Minimizing this loss maximizes the probability the model assigns to the actual text. A loss of 3.0 means the model assigns probability to the correct token on average. State-of-the-art models achieve losses around 1.5-2.0 on web text.
The loss is a maximum likelihood estimator — minimizing it is equivalent to minimizing . With a vocabulary of tokens and context length , the model must assign probability mass across possible sequences. A loss of 2.0 means the model's average per-token surprise is 2 nats, equivalent to choosing among equally likely tokens. Each 0.1 reduction in loss multiplies the probability assigned to correct tokens by — a 10% improvement in confidence.
A model assigns probability 0.8 to the correct next token. What is the per-token loss?
Perplexity is the exponential of the cross-entropy loss. It can be interpreted as the effective number of tokens the model is 'confused' between at each step. A perplexity of 10 means the model is, on average, as uncertain as if choosing uniformly among 10 options. Lower is better. GPT-3 achieved perplexity ~20 on standard benchmarks.
Perplexity converts nats to an effective vocabulary size. If PPL = 20, the model's uncertainty at each step is equivalent to a uniform distribution over 20 tokens. The key insight: perplexity is multiplicative — reducing PPL from 20 to 10 is the same relative improvement as 10 to 5. This is why we compare log-perplexity (cross-entropy) for arithmetic differences. For a vocabulary of , random guessing gives PPL = 50,000; a well-trained model achieving PPL = 15 has reduced uncertainty by a factor of 3,333.
Model A has perplexity 15 on a test set. Model B has perplexity 10. How much better is B?
During training, the model always receives the ground truth previous tokens as input, even if it predicted the wrong token. This is called teacher forcing. It enables parallel training (compute loss for all positions simultaneously) but creates a train-test mismatch: during inference, the model conditions on its own (potentially wrong) predictions.
Teacher forcing computes all positions in parallel: the loss gradient provides a signal at every token simultaneously, giving gradient information per sequence versus for sequence-level methods. The train-test mismatch (exposure bias) grows with generation length: after autoregressive steps, errors compound as in the worst case, where is the single-step error rate. For a 512-token generation, even a 0.1% per-token error rate compounds to ~40% chance of at least one error propagating.
Training sequence: "The cat sat on the mat." If the model predicts "dog" instead of "cat" at position 2, what does it see at position 3?
Standard pretraining uses token-level loss (evaluate each token independently). Sequence-level objectives (e.g., REINFORCE, contrastive learning) evaluate entire generated sequences. Token-level is simpler and more stable for pretraining; sequence-level is used in fine-tuning (RLHF) where quality of the full response matters.
Token-level loss provides gradient signals per sequence (one per position), while sequence-level objectives provide just 1 signal for the entire sequence. The variance of policy gradient estimators scales as for sequence-level rewards (exponential in length), versus for per-token cross-entropy. This is why pretraining uses token-level loss: with and , the search space is — far too large for sequence-level exploration. Sequence-level objectives become practical only after pretraining narrows the distribution to a manageable subspace.
Why not use sequence-level objectives during pretraining?
A model is pretrained with cross-entropy loss and achieves perplexity 12 on a held-out test set. However, when used as a chatbot, it gives poor responses. Why might a low perplexity not translate to good conversational ability?