Chapter 6: Attention Mechanisms

Attention mechanisms solved the fundamental bottleneck of sequence-to-sequence models: compressing an entire input sequence into a single fixed-length vector. Instead of forcing the encoder to pack everything into one representation, attention allows the decoder to dynamically focus on different parts of the input at each generation step. This simple idea --- letting the model learn where to look --- transformed machine translation and laid the groundwork for the Transformer.

The Transformer architecture, introduced in "Attention Is All You Need" (Vaswani et al., 2017), took this further by replacing recurrence entirely with self-attention. Each token in a sequence attends to every other token, computing relevance scores via query-key-value projections. Multi-head attention runs several attention functions in parallel, capturing different types of relationships. Combined with positional encodings, layer normalization, and residual connections, this produces a highly parallelizable architecture that scales to massive datasets.

This chapter traces the evolution from additive (Bahdanau) attention through scaled dot-product self-attention to the full Transformer encoder-decoder. You will also study the major variants --- BERT, GPT, and T5 --- that adapt the Transformer to different tasks by choosing which blocks to keep and how to mask attention.