Loading...
The Transformer Architecture
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
Read the Paper on arXivThe Transformer architecture, introduced in "Attention Is All You Need" (2017), is the foundation of virtually all modern language models (GPT, BERT, LLaMA, etc.) and has since transformed computer vision (ViT), speech, and multimodal AI.
The core insight: self-attention can replace recurrence and convolutions entirely. Instead of processing tokens one-by-one (RNNs) or through local windows (CNNs), self-attention computes relationships between every pair of positions in a single operation — enabling full parallelization and O(1) path length between any two tokens.
The architecture follows an encoder-decoder design for sequence-to-sequence tasks (e.g., translation). Both encoder and decoder are stacks of identical layers, each containing multi-head self-attention, feed-forward networks, and residual connections with layer normalization. The decoder additionally has cross-attention to the encoder output and causal masking to prevent attending to future tokens.
Model configurations from the paper:
| Base | Big | |
|---|---|---|
| Layers (N) | 6 | 6 |
| d_model | 512 | 1024 |
| Heads (h) | 8 | 16 |
| d_k = d_v | 64 | 64 |
| d_ff | 2048 | 4096 |
| Parameters | 65M | 213M |
The base model achieved 27.3 BLEU on English-to-German translation (WMT 2014) and 41.0 BLEU on English-to-French — state-of-the-art at the time — while training in just 3.5 days on 8 GPUs, far faster than any competing model.
Click any topic to jump in
Scaled dot products with $\sqrt{d_k}$ normalization — constant path length between any two tokens.
$h$ parallel attention subspaces at $1/h$ cost each — a cheap ensemble where heads specialize.
Sinusoids at exponential wavelengths — any relative offset is a linear transform of absolute positions.
Cross-attention lets the decoder query any encoder position — no sequential bottleneck.
Position-wise 2-layer MLPs with 4× hidden dimension — key-value memories holding most of the parameters.
Residuals provide a gradient highway; LayerNorm keeps activations stable across depth.
Upper-triangular $-\infty$ mask turns bidirectional attention into a parallelizable language model.
Warmup + Adam + shared dropout — 10× less compute than RNN seq2seq for better BLEU.
Self-attention is the core operation of the Transformer. It allows every token in a sequence to directly attend to every other token, computing context-aware representations in a single matrix operation.
Previous sequence models had fundamental limitations in how they connected distant tokens:
For the sentence "The cat that sat on the mat was hungry," connecting "cat" to "was hungry" (separated by 6 tokens) requires an RNN to propagate information through 6 timesteps. In a Transformer, this connection is direct — computed in a single attention step.
Self-attention computes a weighted combination of all positions, where the weights are learned based on content similarity. Here's exactly how it works:
Step 1: Create Q, K, V matrices
Each input token has an embedding vector (e.g., 512-dimensional). We project this into three separate vectors using learned weight matrices:
For a sequence of tokens: , ,
Step 2: Compute attention scores
Each entry is the dot product between query and key , measuring how much token should attend to token . This produces an matrix — every token compared to every other token.
Step 3: Scale
Why scale? Without scaling, when is large (e.g., 64), dot products grow proportionally to (each is a sum of terms). Large dot products push softmax into regions with extremely small gradients (saturation), making learning near-impossible. Dividing by keeps the variance at ~1 regardless of dimension. The authors specifically tested this: without scaling, performance degraded significantly for large .
Step 4: Softmax → attention weights
Each row sums to 1, creating a probability distribution over which tokens to attend to. High scores become high weights; low scores become near-zero weights.
Step 5: Weighted sum of values
Each output token is a weighted combination of all value vectors, where the weights are the attention probabilities. Token 's output is dominated by the value vectors of tokens it attends to most strongly.
Concrete example with , :
The entire operation is a single matrix multiplication chain — fully parallelizable on GPUs.
Every token attends to every other token in O(1) path length — no information bottleneck
Q, K, V are separate learned projections of the same input: Q = XW^Q, K = XW^K, V = XW^V. Queries ask "what am I looking for?", keys advertise "what do I contain?", values provide "what information do I give?"
Scaling by is critical — without it, softmax saturates for large and gradients vanish. The paper empirically confirmed this degradation
Computational complexity is — quadratic in sequence length. This is the main bottleneck for long sequences (addressed by later work like FlashAttention, Linformer, etc.)
Self-attention is permutation-equivariant: reordering inputs produces the same reordered outputs. This is why positional encodings are necessary
Unlike convolutions, attention has no inductive bias for locality — it must learn all spatial relationships from data, which requires more training data but is ultimately more flexible
Q ∈ ℝ^(n×d_k), K ∈ ℝ^(n×d_k), V ∈ ℝ^(n×d_v). The output is ℝ^(n×d_v). Each row of the output is a weighted average of V rows, where weights come from softmax of scaled dot products between Q and K rows.
Both have similar theoretical power, but dot-product attention is much faster in practice because it can be implemented with optimized matrix multiplication (BLAS). Additive attention requires a feed-forward network per pair.
. The division by prevents dot products from growing with dimension: for random with unit variance, , so softmax would saturate without the scale. Complexity is — constant path length between any two tokens, unlike RNNs which need sequential steps.