Chapter 5: Sequence Models

Along the cell path, $\partial C_t / \partial C_{t-1} = \text{diag}(f_t)$, so a gradient crossing $k$ steps is scaled by the product of the forget values. With $f = 0.98$ at every step, 100 steps keep $0.98^{100} \approx 0.13$ of the signal; with $f = 0.5$, ten steps keep only about 0.001. The gate is a soft value in $(0, 1)$, not a binary switch. A common practice from Gers et al. (2000), recommended again by Jozefowicz et al. (2015), initializes the forget bias $b_f$ to 1, so $f_t = \sigma(1) \approx 0.73$ before training, which biases a new network toward remembering rather than forgetting.

The input gate $i_t = \sigma(W_i [h_{t-1}, x_t] + b_i)$ and candidate $\tilde{C}_t = \tanh(W_C [h_{t-1}, x_t] + b_C)$ work as a write mechanism. The tanh produces candidate values in $[-1, 1]$ (what to write), while the sigmoid produces a gating mask in $(0, 1)$ (how much to write). Their element-wise product $i_t \odot \tilde{C}_t$ means each cell dimension independently decides both the content and the strength of the update. This separation of content generation from write control is what makes LSTMs able to precisely accumulate information over long sequences.

The update $C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t$ is additive rather than multiplicative. This is the critical difference from vanilla RNNs where $h_t = \tanh(Wh_{t-1} + \ldots)$ involves a matrix multiply through $W$. Addition means the gradient $\partial C_t / \partial C_{t-1} = f_t$ is a diagonal matrix (just the forget gate values), not a full matrix multiply. Over $T$ steps, the gradient is $\prod_k f_k$ — a product of values in $(0,1)$, which can be kept near 1 by learning appropriate biases, unlike the uncontrolled spectral-norm products in vanilla RNNs.

The output gate $o_t = \sigma(W_o [h_{t-1}, x_t] + b_o)$ controls how much of the cell state is exposed to the rest of the network via $h_t = o_t \odot \tanh(C_t)$. The tanh squashes the cell state (which can accumulate to large values) back to $[-1, 1]$, and the sigmoid selectively reveals dimensions. This means the LSTM maintains two distinct representations: the cell state $C_t$ for long-term storage (unbounded, additive updates) and the hidden state $h_t$ for short-term computation (bounded, gated exposure).

The update gate $z_t = \sigma(W_z [h_{t-1}, x_t])$ interpolates between the previous hidden state and the candidate: $h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$. When $z_t \approx 0$, the hidden state is copied forward unchanged ($h_t \approx h_{t-1}$), creating the same gradient highway as the LSTM forget gate. When $z_t \approx 1$, the state is fully replaced. This convex combination with learned coefficients subsumes the LSTM's separate forget and input gates into a single mechanism that enforces $f_t + i_t = 1$.

The reset gate $r_t = \sigma(W_r [h_{t-1}, x_t])$ controls how much of the previous hidden state contributes to the candidate: $\tilde{h}_t = \tanh(W[r_t \odot h_{t-1}, x_t])$. When $r_t \approx 0$, the candidate ignores the past entirely and acts like a feedforward network on $x_t$ alone. This allows the GRU to make sharp transitions — completely resetting its memory when the context changes dramatically (e.g., a sentence boundary). The reset gate operates on $h_{t-1}$ before it enters the candidate computation, while the update gate operates on the output — a two-stage gating pipeline.

The candidate $\tilde{h}_t = \tanh(W_h [r_t \odot h_{t-1}, x_t] + b_h)$ is the proposed new content, computed from a reset-gated version of the previous state. The tanh bounds the candidate to $[-1, 1]$, and the final state is a convex combination $(1-z_t)h_{t-1} + z_t \tilde{h}_t$. With $d$ hidden dimensions, the GRU has $3d(d + d_x)$ parameters (three gate/candidate matrices) versus the LSTM's $4d(d + d_x)$ (four gates). This 25% parameter reduction often yields comparable performance with faster training, especially on smaller datasets where the LSTM's extra capacity can overfit.

The forward RNN computes $\overrightarrow{h}_t = f(x_t, \overrightarrow{h}_{t-1})$ left-to-right and the backward RNN computes $\overleftarrow{h}_t = f(x_t, \overleftarrow{h}_{t+1})$ right-to-left. At position $t$, the forward state summarizes $x_1, \ldots, x_t$ and the backward state summarizes $x_T, \ldots, x_t$. Together they provide the complete sequence context $x_1, \ldots, x_T$ at every position. The two RNNs have independent parameters — they do not share weights — so the model has exactly twice the parameters of a unidirectional RNN.

The concatenated representation $h_t = [\overrightarrow{h}_t; \overleftarrow{h}_t]$ has dimension $2d$ where $d$ is the hidden size of each direction. This doubles the representation capacity at each position. When projecting to an output space of dimension $k$, the projection matrix is $W \in \mathbb{R}^{2d \times k}$, meaning the model can learn to weight forward and backward context differently for each output dimension. For tasks like NER where both left and right context are informative ("Dr. [PERSON] visited"), the bidirectional representation captures signals that a unidirectional model would miss from one direction.

Bidirectional RNNs require the full input sequence to be available before computation — the backward pass starts from the last token. This makes them unsuitable for autoregressive generation where tokens are produced one at a time, because future tokens do not exist yet. However, for encoding tasks (classification, tagging, encoding in seq2seq), bidirectionality provides strictly more information at each position. The computational cost is exactly $2\times$ a unidirectional RNN, but the two directions can be computed in parallel on separate hardware.

The encoder maps a variable-length input $(x_1, \ldots, x_T)$ to a fixed-size context $c = h_T \in \mathbb{R}^d$. For a long input this is a lossy compression: the same $d$ numbers must summarize 5 tokens or 500, and the recurrent update overwrites early information as new tokens arrive. Real-valued vectors could in principle store a great deal, so the practical limit is what gradient descent can learn to pack into $h_T$, not raw bit capacity. Sutskever et al. found that reversing the source sentence improved results, because it shortened the distance between the first source words and the first target words. Attention, in Chapter 6, removes the bottleneck.

The decoder generates tokens autoregressively: $P(y_t \mid y_{<t}, c) = \text{softmax}(W_s h_t^{\text{dec}})$ where $h_t^{\text{dec}} = f(y_{t-1}, h_{t-1}^{\text{dec}}, c)$. At inference time, each predicted token is fed back as input to produce the next token. This creates a sequential dependency chain where errors compound — a wrong token at step $t$ shifts the entire conditional distribution for all subsequent tokens, a phenomenon known as exposure bias.

During training, teacher forcing feeds the ground-truth token $y_{t-1}^*$ (not the model's prediction $\hat{y}_{t-1}$) as input at each step. This means the loss at step $t$ is $-\log P(y_t^* \mid y_{<t}^*, c)$ — conditioned on perfect history. The discrepancy is that at inference time, the model conditions on its own (possibly incorrect) predictions. This train-test mismatch means the model never learns to recover from its own errors. Scheduled sampling addresses this by randomly replacing ground-truth inputs with model predictions during training, with the replacement probability increasing over the course of training.

The context vector $c \in \mathbb{R}^d$ must encode all source information regardless of sequence length. For a sentence of $T$ tokens each with embedding dimension $d_e$, the input has $T \cdot d_e$ dimensions of information compressed into $d$ dimensions. When $T \cdot d_e \gg d$, significant information loss is inevitable. Empirically, BLEU scores for seq2seq translation drop sharply for sentences longer than 20--30 words, precisely because the bottleneck cannot represent the fine-grained lexical and structural details needed for accurate translation of long sentences.

The simplest context vector is $c = h_T^{\text{enc}}$, the encoder's final hidden state. A common improvement is $c = [\overrightarrow{h}_T; \overleftarrow{h}_1]$, concatenating the final states of a bidirectional encoder, which captures information processed from both ends. Another option is $c = \text{mean}(h_1, \ldots, h_T)$ or $c = \max(h_1, \ldots, h_T)$, pooling across all time steps. Each design trades off between compression (more lossy) and coverage (more complete). The attention mechanism (Chapter 6) eliminates this trade-off entirely by computing a different weighted combination of all encoder states at each decoding step.

Beam search maintains $k$ hypotheses (beams) at each decoding step, expanding each by the top-$V$ vocabulary tokens and keeping the $k$ highest-scoring partial sequences. The score of a hypothesis is $\log P(y_1, \ldots, y_t) = \sum_{i=1}^{t} \log P(y_i \mid y_{<i})$. With beam width $k$, the algorithm explores $O(k \cdot V)$ candidates per step, making it $k$ times more expensive than greedy decoding but dramatically improving output quality. Length normalization divides the log-probability by $t^\alpha$ (typically $\alpha = 0.6$) to prevent the search from favoring shorter sequences, whose cumulative log-probabilities are less negative.

Nucleus sampling selects from the smallest set of tokens whose cumulative probability exceeds a threshold $p$: $\{w : \sum_{w' \in V_p} P(w') \geq p\}$ where $V_p$ is sorted by descending probability. Unlike top-$k$ sampling which always keeps exactly $k$ tokens, nucleus sampling adapts the candidate set size to the model's confidence. When the model is confident (one token has probability 0.9), the nucleus contains just 1--2 tokens. When uncertain (flat distribution), it may contain dozens. This adaptive behavior produces more natural and diverse text than fixed-$k$ sampling.