Chapter 5: Sequence Models
Master recurrent neural networks and their variants for processing sequential text data — from vanilla RNNs to LSTMs, GRUs, and the encoder-decoder paradigm.
Chapter Overview
Sequence models are the backbone of modern NLP systems that process text as ordered sequences of tokens. Unlike feedforward networks that treat each input independently, recurrent architectures maintain a hidden state that accumulates information as it reads through a sequence one token at a time. This memory mechanism allows the model to capture dependencies between words regardless of their distance in the sentence.
The vanilla RNN introduced the fundamental idea of recurrence but suffers from the vanishing gradient problem, which makes it difficult to learn long-range dependencies. The Long Short-Term Memory (LSTM) network and the Gated Recurrent Unit (GRU) solve this with gating mechanisms that control information flow through the network. These gated architectures became the dominant approach for tasks like language modeling, machine translation, and text classification throughout the 2010s.
This chapter traces the evolution from simple RNNs through gated variants, bidirectional processing, and the encoder-decoder framework that enabled sequence-to-sequence tasks like translation. Understanding these architectures is essential background for the attention mechanism (Chapter 6) and the Transformer (Chapter 7), which ultimately replaced recurrence with parallelizable self-attention.
Chapter Roadmap
Click any topic to jump in
Recurrent Neural Networks
The fundamental recurrence equation, BPTT, and why vanilla RNNs struggle with long-range dependencies.
Two gated architectures that preserve long-range gradients
LSTM
Gating mechanisms that solve the vanishing gradient problem via an additive cell state highway.
GRU
A streamlined alternative to LSTM that merges the cell and hidden state with fewer gates.
Bidirectional RNNs
Running two RNNs in opposite directions to capture both past and future context at every position.
Encoder compresses, decoder generates with search strategies
Sequence-to-Sequence
The encoder-decoder paradigm for mapping variable-length input sequences to variable-length outputs.
Encoder-Decoder Architecture
Context vector design, beam search decoding, and nucleus sampling for generation quality.
Sign up to unlock this chapter
This chapter is part of PixelBank Premium. Create a free account, then upgrade to read the full lesson — concepts, walkthroughs, and exercises.