Attention Is All You Need

Previous sequence models had fundamental limitations in how they connected distant tokens:

• RNNs/LSTMs: Process tokens left-to-right sequentially. To connect token 1 to token 100, information must flow through 99 intermediate steps — causing vanishing gradients and making parallelization impossible. Training time grows linearly with sequence length.
• CNNs (ConvS2S): Use local receptive fields (e.g., kernel size 3). Connecting distant tokens requires $O(n/k)$ layers of convolutions stacked on top of each other, where $k$ is the kernel width. A 512-token sequence with kernel 5 needs ~100 layers.
• Both approaches create a bottleneck: the longer the sequence, the harder it is to learn long-range dependencies.

For the sentence "The cat that sat on the mat was hungry," connecting "cat" to "was hungry" (separated by 6 tokens) requires an RNN to propagate information through 6 timesteps. In a Transformer, this connection is direct — computed in a single attention step.

Self-attention computes a weighted combination of all positions, where the weights are learned based on content similarity. Here's exactly how it works:

Step 1: Create Q, K, V matrices

Each input token has an embedding vector $x_i \in \mathbb{R}^{d_{model}}$ (e.g., 512-dimensional). We project this into three separate vectors using learned weight matrices:

• Query: $Q = XW^Q$ where $W^Q \in \mathbb{R}^{d_{model} \times d_k}$ (e.g., 512 × 64)
• Key: $K = XW^K$ where $W^K \in \mathbb{R}^{d_{model} \times d_k}$ (e.g., 512 × 64)
• Value: $V = XW^V$ where $W^V \in \mathbb{R}^{d_{model} \times d_v}$ (e.g., 512 × 64)

For a sequence of $n$ tokens: $X \in \mathbb{R}^{n \times d_{model}}$, $Q, K \in \mathbb{R}^{n \times d_k}$, $V \in \mathbb{R}^{n \times d_v}$

Step 2: Compute attention scores

$\text{scores} = QK^T \in \mathbb{R}^{n \times n}$

Each entry $\text{scores}[i][j]$ is the dot product between query $i$ and key $j$, measuring how much token $i$ should attend to token $j$. This produces an $n \times n$ matrix — every token compared to every other token.

Step 3: Scale

$\text{scaled\_scores} = \frac{QK^T}{\sqrt{d_k}}$

Why scale? Without scaling, when $d_k$ is large (e.g., 64), dot products grow proportionally to $d_k$ (each is a sum of $d_k$ terms). Large dot products push softmax into regions with extremely small gradients (saturation), making learning near-impossible. Dividing by $\sqrt{d_k}$ keeps the variance at ~1 regardless of dimension. The authors specifically tested this: without scaling, performance degraded significantly for large $d_k$.

Step 4: Softmax → attention weights

$\alpha = \text{softmax}(\text{scaled\_scores})$

Each row sums to 1, creating a probability distribution over which tokens to attend to. High scores become high weights; low scores become near-zero weights.

Step 5: Weighted sum of values

$\text{output} = \alpha V \in \mathbb{R}^{n \times d_v}$

Each output token is a weighted combination of all value vectors, where the weights are the attention probabilities. Token $i$'s output is dominated by the value vectors of tokens it attends to most strongly.

Concrete example with $n = 4$, $d_k = 64$:

• Input: 4 tokens, each 512-dim → $X \in \mathbb{R}^{4 \times 512}$
• After projection: $Q, K, V \in \mathbb{R}^{4 \times 64}$
• Scores: $QK^T \in \mathbb{R}^{4 \times 4}$ (16 pairwise similarities)
• After softmax: $\alpha \in \mathbb{R}^{4 \times 4}$ (each row sums to 1)
• Output: $\alpha V \in \mathbb{R}^{4 \times 64}$

The entire operation is a single matrix multiplication chain — fully parallelizable on GPUs.