Chapter 9: Language Modeling

Language modeling is the task of assigning probabilities to sequences of words. Given a sequence $w_1, w_2, \ldots, w_{n-1}$, a language model estimates $P(w_n \mid w_1, \ldots, w_{n-1})$ --- the probability of the next word given all preceding context. This deceptively simple objective underlies virtually every modern NLP system: machine translation, speech recognition, text generation, and even code completion all rely on language models to score or generate fluent text.

The history of language modeling traces a path from counting word co-occurrences to learning distributed representations. Statistical n-gram models dominated for decades, approximating context with fixed-length windows and smoothing techniques to handle unseen word combinations. Neural language models, starting with Bengio's feedforward architecture in 2003, replaced discrete counts with continuous vector representations, enabling generalization across similar contexts. Recurrent neural networks extended the context window to arbitrary lengths, and the transformer architecture eliminated sequential bottlenecks entirely.

Today, two paradigms define the field. BERT's masked language modeling (MLM) trains bidirectional representations by predicting randomly masked tokens from surrounding context in both directions. GPT's causal language modeling (CLM) trains left-to-right autoregressive models that generate text one token at a time. These two approaches --- predicting masked tokens versus predicting next tokens --- lead to fundamentally different model architectures and downstream capabilities. This chapter covers the full trajectory from n-grams to transformers, the mathematics of perplexity as an evaluation metric, and the architectural choices that distinguish BERT from GPT.