Chapter 5: Sequence Models

Sequence models are the backbone of modern NLP systems that process text as ordered sequences of tokens. Unlike feedforward networks that treat each input independently, recurrent architectures maintain a hidden state that accumulates information as it reads through a sequence one token at a time. This memory mechanism allows the model to capture dependencies between words regardless of their distance in the sentence.

The vanilla RNN introduced the fundamental idea of recurrence but suffers from the vanishing gradient problem, which makes it difficult to learn long-range dependencies. The Long Short-Term Memory (LSTM) network and the Gated Recurrent Unit (GRU) solve this with gating mechanisms that control information flow through the network. These gated architectures became the dominant approach for tasks like language modeling, machine translation, and text classification throughout the 2010s.

This chapter traces the evolution from simple RNNs through gated variants, bidirectional processing, and the encoder-decoder framework that enabled sequence-to-sequence tasks like translation. Understanding these architectures is essential background for the attention mechanism (Chapter 6) and the Transformer (Chapter 7), which ultimately replaced recurrence with parallelizable self-attention.