Loading...
An Image is Worth 16x16 Words
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby
Read the Paper on arXivVision Transformer (ViT) demonstrated that a pure Transformer architecture, applied directly to sequences of image patches, can match or exceed state-of-the-art CNNs on image classification -- without any convolutions. The key insight is that an image can be treated as a sequence of flattened 2D patches, analogous to tokens (words) in NLP.
The architecture is remarkably simple: split a image into a grid of patches, producing patches. Each patch is flattened to a 768-dimensional vector and linearly projected to the model dimension . A learnable [CLS] token is prepended (making 197 tokens total), learnable 1D position embeddings are added, and the entire sequence is fed to a standard Transformer encoder with multi-head self-attention and MLP blocks. The [CLS] token's output representation is passed through a classification head.
The critical finding: ViT's performance depends heavily on pre-training data scale. Trained only on ImageNet-1K (1.2M images), ViT-B/16 achieves just 77.9% top-1 -- below ResNet-152's 78.3%. But pre-trained on JFT-300M (300M images), ViT-L/16 reaches 87.8% top-1 on ImageNet, surpassing the best CNNs while using 4x fewer FLOPs to train. ViT-H/14 pre-trained on JFT-300M achieves 88.6% top-1, setting a new state-of-the-art. This revealed a fundamental tradeoff: Transformers lack the inductive biases of CNNs (locality, translation equivariance) but compensate with superior scalability when sufficient data is available.
Click any topic to jump in
16×16 patches flattened and linearly projected — equivalent to one stride-16 convolution.
Learnable per-patch vectors added to embeddings — recover 2D spatial topology from scratch via gradient descent.
A learnable global-reader token aggregates information from all patches — adaptive pooling via attention weights.
Global receptive field from layer 1 — but quadratic cost and no translation bias means more data is needed.
ViT underperforms CNNs on small datasets but overtakes them beyond ~100M images — lower bias, higher variance.
Distillation (DeiT) and masked reconstruction (BEiT/MAE) make ViT work without massive labeled datasets.
Attention rollout reveals that ViT discovers object boundaries without segmentation supervision.
Images are split into non-overlapping patches, which are then linearly projected to create token embeddings -- treating each patch as a 'visual word' in a sequence. This is the mechanism that bridges the gap between 2D images and the 1D sequence input that Transformers expect.
Transformers were designed for 1D sequences (text), and naively applying self-attention to raw pixels is computationally impossible:
Patch Embedding converts a 2D image into a manageable 1D sequence by treating non-overlapping image regions as tokens:
Step-by-step for a standard input with patch size :
Split into patches: Divide the image into a grid of non-overlapping patches. Each patch is pixels.
Flatten each patch: Reshape each patch into a vector of dimension .
Linear projection: Multiply each 768-dim vector by a learnable projection matrix to get the token embedding. For ViT-Base, , so is (589K parameters). For ViT-Large, , so is (786K parameters).
Result: A sequence of 196 token embeddings, each of dimension . The sequence length is now 256x smaller than the pixel count (196 vs 50,176).
Implementation detail: The linear projection is equivalent to a 2D convolution with kernel size = stride = . In practice, ViT uses nn.Conv2d(3, D, kernel_size=16, stride=16) which produces a feature map, then reshapes to .
Patch size is the critical tradeoff:
| Patch Size | Tokens | Token Dim | Attention Cost | Detail |
|---|---|---|---|---|
| 32x32 | 49 | 3072 | Low | Coarse |
| 16x16 | 196 | 768 | Medium | Standard |
| 14x14 | 256 | 588 | Higher | Fine |
| 8x8 | 784 | 192 | Very High | Very Fine |
Smaller patches capture finer spatial detail but increase the sequence length quadratically, making attention much more expensive. ViT-B/16 (patch size 16) and ViT-L/16 are the standard configurations. ViT-H/14 uses patch size 14 for slightly finer resolution at higher compute cost.
Patch size is the most important hyperparameter: it determines both sequence length () and computational cost ( attention)
For with : each patch is values, projected to model dimension via a learnable linear layer
The projection is implemented as Conv2d(3, D, kernel_size=P, stride=P) -- mathematically equivalent to flatten + linear, but more efficient on GPU
Sequence length reduces from 50,176 (pixels) to 196 (patches) -- a 256x reduction that makes self-attention tractable
Unlike CNNs, patches have no overlap and no hierarchy -- all spatial information mixing happens through self-attention across the full sequence
Patch embedding has only parameters for ViT-Base -- a tiny fraction of the model's total 86M parameters
x_p^i is the flattened i-th patch (dim P²C), E is the projection matrix (dim P²C × D), x_class is the prepended [CLS] token, and E_pos are the position embeddings. This produces the full input sequence of N+1 tokens, each of dimension D.
For 224x224 with P=16: N = 196. For 224x224 with P=14: N = 256. For 384x384 with P=16: N = 576 (used during fine-tuning at higher resolution).
A image is cut into patches, giving tokens. Each patch is flattened and linearly projected: with . This is mathematically equivalent to a single stride-16 conv with 768 input channels — so ViT's 'first layer' is a CNN layer in disguise, just with no spatial overlap.