Chapter 10: CNNs & Sequence Models

Max pooling slides a window, usually 2 by 2 with stride 2, over each channel and keeps the largest value in each window, halving height and width. Because a convolutional feature map records how strongly a pattern appears at each location, the maximum answers whether the pattern appears anywhere in this window. Small shifts of the input often leave the maximum unchanged, so max pooling adds some local translation invariance, though not much. In backpropagation, the whole gradient flows to the winning position and the others get zero. It has no parameters, and it was the standard downsampling step in CNNs from LeNet through VGG.

Average pooling replaces each window with the mean of its values, so every activation contributes equally. The output is smoother and keeps information about overall activation strength, but a single strong response gets diluted by weak neighbors, so it is less selective than max pooling. In backpropagation, the gradient is split equally among all positions in the window, one quarter each for 2 by 2. Average pooling was used in LeNet-5 and is mostly found today in its global form at the end of a network. Within the body of a network, max pooling or strided convolution usually works better for detecting whether a feature is present.

Global average pooling (GAP) averages each channel over its entire spatial extent, turning a $C \times H \times W$ tensor into a vector of $C$ numbers, one per feature detector. Introduced in Network in Network (2013), it replaces flattening followed by large dense layers. A final linear layer then maps the $C$ values to class scores. GAP has no parameters, removes most of the classifier head's weights, and acts as a structural regularizer. It also lets one network accept different input sizes, because the output length depends only on the channel count. ResNet, DenseNet, and EfficientNet all end with GAP. The cost is discarding where features appear.

There are two ways to downsample: a pooling layer, or a convolution with stride 2. A strided convolution computes its output only at every second position, so it halves the spatial size while learning how to summarize each region, instead of applying a fixed max or mean. The All Convolutional Net (2014) showed that replacing max pooling with strided convolutions can match accuracy with a simpler architecture. Most modern CNNs downsample mainly with strided convolutions, though ResNet still applies one max pooling after its first layer and ends with global average pooling. The trade-off is parameters and computation, since pooling is free and a strided convolution is not.

At each step, the RNN cell combines the previous hidden state $h_{t-1}$ and the current input $x_t$: multiply each by its weight matrix, $W_{hh}$ and $W_{xh}$, add the bias $b$, and apply tanh. The same weights are used at every step, so the parameter count does not depend on sequence length. Training unrolls the loop into a deep network with one layer per step and backpropagates through time. Gradients reaching early steps are multiplied by $W_{hh}$ and a tanh derivative once per step, so over long sequences they usually vanish, or with large weights explode. Gradient clipping fixes explosion, but not vanishing.

The LSTM, from Hochreiter and Schmidhuber (1997), adds a cell state $c_t$ alongside the hidden state, updated additively: $c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$. Three sigmoid gates, each computed from $h_{t-1}$ and $x_t$, control the flow: the forget gate $f_t$ decides how much old memory to keep, the input gate $i_t$ how much of the new candidate $\tilde{c}_t$ to write, and the output gate how much of the cell to expose as $h_t$. Along the cell path, the gradient is multiplied by the forget gate, not by a weight matrix and squashing derivative, so it survives many steps when the gate stays near 1.

The GRU, from Cho et al. (2014), simplifies the LSTM. It merges the cell and hidden states and uses two gates instead of three. The update gate $z_t$ interpolates between keeping the old state and writing a new candidate, $h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$, playing the forget and input roles at once. The reset gate controls how much of the old state feeds into the candidate. With three weight blocks instead of the LSTM's four, a GRU has about three quarters of the parameters and runs faster. Comparative studies find neither consistently better, so the choice is empirical.

A standard RNN's state at step $t$ summarizes only the inputs up to $t$, but the meaning of a word often depends on what follows it. A bidirectional RNN runs two independent RNNs, one left to right and one right to left, and concatenates their hidden states at each position, so every position's representation reflects the whole sequence. This doubles the parameters and state size. It suits tasks where the full input is available before any output is needed, such as named entity recognition, tagging, or encoding a sentence for translation. It cannot be used for generation or real-time streaming, where future inputs do not exist yet.

Attention is a soft lookup. Each query vector is compared with every key by a dot product, measuring how relevant that position is. The scores are divided by $\sqrt{d_k}$, the key dimension, and a softmax turns them into weights that sum to 1. The output is the weighted average of the value vectors. The scaling matters because dot products of random $d_k$-dimensional vectors have standard deviation $\sqrt{d_k}$; unscaled, large scores push the softmax toward one-hot outputs with tiny gradients. Every position attends to every other in one step, so path length is constant, but computing all pairs costs time and memory that grow quadratically with sequence length.

A single attention operation produces one weighted average per position, so it can capture only one kind of relationship at a time. Multi-head attention runs $h$ attention heads in parallel. Each head projects the input into its own smaller query, key, and value space of dimension $d_k = d_{model}/h$, attends there, and the $h$ outputs are concatenated and mixed by an output projection. Because the head dimension shrinks as heads are added, the total cost stays about the same as one full-width head. With four $d_{model} \times d_{model}$ projections, the layer has about $4d_{model}^2$ parameters. Heads often specialize, but many prove redundant and can be pruned.

A Transformer layer has two sublayers: multi-head self-attention, which mixes information across positions, and a position-wise feedforward network, two linear layers with a hidden size usually four times $d_{model}$, applied to each position separately. Each sublayer is wrapped in a residual connection and layer normalization. The original model had an encoder stack and a decoder stack, with cross-attention from decoder to encoder. Self-attention alone has no notion of order: permuting the inputs permutes the outputs in the same way. So positional encodings, fixed sinusoids or learned vectors, are added to the input embeddings. Removing recurrence lets all positions train in parallel.

Transformers are usually pre-trained on huge unlabeled text corpora, then fine-tuned or prompted for specific tasks. Three families exist. Encoder-only models such as BERT use bidirectional self-attention and are pre-trained by masked language modeling, predicting about 15 percent of hidden tokens; they suit classification, tagging, and retrieval. Decoder-only models such as GPT use a causal mask, so each position sees only earlier tokens, and are trained to predict the next token; they suit generation and now handle most tasks through prompting. Encoder-decoder models such as T5 suit sequence-to-sequence tasks like translation and summarization. Vision Transformers apply the same encoder to sequences of image patches.