Vision Transformer (ViT)

Self-attention computes: $\text{Attn}(Q, K, V) = \text{softmax}(QK^T/\sqrt{d})V$

This operation is permutation-equivariant: if you permute the input tokens, the output tokens are permuted in exactly the same way. Mathematically: $\text{Attn}(\Pi Q, \Pi K, \Pi V) = \Pi \cdot \text{Attn}(Q, K, V)$

where $\Pi$ is any permutation matrix. This means:

• Shuffling all 196 patches randomly would produce the same set of output representations (just reordered) -- the model literally cannot tell which patch was top-left vs bottom-right.
• Spatial relationships (adjacency, relative position, global location) are completely invisible to the Transformer.
• An image of a face with eyes swapped to the chin area would be processed identically to the normal face, unless position information is injected.
• In NLP, this same problem exists and is solved with position embeddings. For vision, the 2D spatial structure makes the problem even more acute since patches have both row and column coordinates.

Learned 1D Position Embeddings are added element-wise to patch embeddings before the Transformer encoder:

$\mathbf{z}_0^i = \mathbf{x}_p^i E + \mathbf{e}_{\text{pos}}^i \quad \text{for } i = 0, 1, \ldots, N$

where $\mathbf{e}_{\text{pos}}^i \in \mathbb{R}^D$ is a learnable vector for position $i$ (including position 0 for the [CLS] token).

Key design decisions and findings:

1. 1D vs 2D position embeddings: The paper tested both. 1D: a single index $i \in \{0, 1, \ldots, 196\}$ with raster-scan order. 2D: separate row and column embeddings $(r, c)$ concatenated. Result: no significant difference (<0.1% accuracy). The model learns the 2D grid structure from 1D positions on its own.

2. Learned vs sinusoidal: Original Transformer (Vaswani et al.) used fixed sinusoidal encodings. ViT uses fully learned embeddings. Result: no significant difference for ViT. Both work equally well, likely because the fixed patch grid is simpler than variable-length text.

3. What the learned embeddings look like: After training, the position embedding similarity matrix reveals clear 2D spatial structure. Position $i$ has high cosine similarity with positions in the same row and column, and similarity falls off with spatial distance. The model recovers the 2D grid purely from data.

4. Parameter count: For ViT-Base with 196 + 1 = 197 positions and $D = 768$: the position embedding table has $197 \times 768 = 151{,}296$ parameters -- negligible compared to the 86M total.

Resolution transfer and position embedding interpolation:

ViT is pre-trained at $224 \times 224$ (196 patches) but often fine-tuned at higher resolution (e.g., $384 \times 384$, giving $576$ patches). Since the learned position embeddings are for 196 positions, they must be interpolated to 576 positions using 2D bicubic interpolation: the $14 \times 14$ position embedding grid is upsampled to $24 \times 24$. This works well because the learned embeddings encode smooth spatial functions.

After the Transformer encoder processes the sequence, we have $N + 1$ output token representations (one per patch plus [CLS]). We need to reduce these to a single vector for classification. Several approaches exist:

1. Global average pooling (GAP) over all $N$ patch tokens: $\mathbf{y} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{z}_L^i$. Simple but treats all patches equally -- background patches contribute as much as object patches.
2. Use a specific patch token: Arbitrary and biased toward that spatial location.
3. Learnable aggregation token ([CLS]): A dedicated token that learns to aggregate task-relevant information from all patches through attention.

ViT follows BERT's approach from NLP, where a [CLS] token at position 0 serves as the sequence-level representation. This design choice was made to stay as close to the standard NLP Transformer as possible, validating that the same architecture transfers to vision with minimal modification.

Interestingly, the paper found that GAP works comparably to [CLS] when combined with appropriate learning rate tuning. The choice of [CLS] is primarily about architectural consistency with NLP Transformers rather than a clear accuracy advantage.

The [CLS] token mechanism:

1. Initialization: A learnable vector $\mathbf{x}_{\text{class}} \in \mathbb{R}^D$ is initialized randomly (typically from $\mathcal{N}(0, 0.02)$). For ViT-Base, this is a single 768-dimensional vector -- just 768 parameters.

2. Prepending: It becomes position 0 in the input sequence: $\mathbf{z}_0 = [\mathbf{x}_{\text{class}};\ \mathbf{x}_p^1 E;\ \cdots;\ \mathbf{x}_p^N E] + \mathbf{E}_{\text{pos}}$

3. Attention-based aggregation: Over $L$ Transformer layers, [CLS] participates in self-attention with all 196 patch tokens. At each layer, it can attend to any patch and any patch can attend to it. After 12 layers (ViT-Base), [CLS] has been iteratively refined by aggregating information from the entire image.

4. Classification head: The final [CLS] output $\mathbf{z}_L^0$ is fed through an MLP head:

   • During pre-training: $\text{MLP}(\mathbf{z}_L^0)$ with one hidden layer and tanh activation
   • During fine-tuning: a single linear layer $W \mathbf{z}_L^0 + b$ mapping $D \to K$ classes

5. Only [CLS] is used: The 196 patch output tokens are discarded for classification (though they are useful for other tasks like detection or segmentation).

How [CLS] learns to aggregate: In early layers, [CLS] attends broadly to many patches, building a coarse global representation. In later layers, attention becomes more focused on discriminative regions (e.g., the subject of the image). This progressive refinement is analogous to how CNN features go from edges to textures to object parts across layers.

CNNs encode two strong inductive biases that constrain how they process spatial information:

1. Locality: A 3x3 convolution kernel only sees a $3 \times 3 = 9$ pixel neighborhood. To get a receptive field of $r$ pixels, you need roughly $r/2$ layers of 3x3 convolutions. For a 224x224 image, global context requires ~112 layers -- far deeper than most practical networks.

2. Translation equivariance: The same filter is applied at every spatial position (weight sharing). A cat detector learned in the top-left works identically in the bottom-right. This is a powerful bias for natural images but also a constraint.

Practical consequences:

• In ResNet-50, the effective receptive field at conv5 is roughly 128x128 pixels -- still not global for 224x224 inputs. Information from opposite corners never directly interacts until the global average pooling at the very end.
• Long-range dependencies (e.g., matching a face to body pose) require the gradient signal to propagate through many layers, making these relationships harder to learn.
• The hierarchical feature pyramid (low-level $\to$ high-level) is encoded in the architecture itself: early layers MUST be local because the kernel size constrains them.

ViT self-attention is global from layer 1:

At every Transformer layer, the self-attention computation is: $\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V$

where $Q, K, V \in \mathbb{R}^{(N+1) \times d_k}$ are the query, key, and value projections of all $N+1$ tokens. The attention matrix $A = \text{softmax}(QK^T/\sqrt{d_k}) \in \mathbb{R}^{(N+1) \times (N+1)}$ has entries for every pair of tokens -- patch 1 in the top-left can directly attend to patch 196 in the bottom-right.

What ViT gains from global attention:

• Immediate long-range dependencies: A patch containing an eye can directly reference a patch containing a nose in the first layer, without needing dozens of intermediate layers.
• Flexible receptive field: Each attention head can learn its own pattern -- some heads attend locally (mimicking convolutions), others attend globally, and patterns change across layers.
• No feature hierarchy constraint: All layers operate at the same resolution ($N+1$ tokens of dimension $D$), so high-level reasoning can happen at any layer.

What ViT loses (the inductive bias gap):

• No locality bias: ViT must learn from data that nearby patches are more related than distant ones. CNNs get this for free from kernel locality.
• No translation equivariance: A feature learned at one position is not automatically applicable at another. ViT must learn shift invariance from data augmentation and diverse training examples.
• No built-in hierarchy: CNNs progressively reduce resolution and increase channels, creating an automatic multi-scale representation. ViT processes all tokens at the same resolution.
• Quadratic cost: Attention is $O(N^2 D)$ vs convolution's $O(N k^2 D)$. For $N = 196$ and $k = 3$, attention is $196/9 \approx 22\times$ more expensive per layer.

The key tradeoff: CNNs use architectural bias to compensate for limited data. ViT uses massive data to compensate for missing architectural bias. Above ~100M training images, ViT's flexibility becomes an advantage; below that, CNN's inductive biases win.

ViT has almost no built-in inductive biases for vision:

• No locality: Self-attention connects all patches globally. The model must learn that spatially adjacent patches tend to be semantically related -- a fact CNNs encode for free via local kernels.
• No translation equivariance: A filter that detects a horizontal edge at position (3, 5) does not automatically detect it at (10, 15). ViT must see the same pattern at many positions during training to generalize.
• No hierarchical processing: CNNs create multi-scale features (edges $\to$ textures $\to$ parts $\to$ objects) through their progressive downsampling architecture. ViT processes all 196 tokens at the same resolution, requiring it to learn multi-scale representation internally.

The consequence is dramatic on small datasets:

• On ImageNet-1K alone (1.2M images), ViT-B/16 achieves 77.9% top-1 -- significantly below ResNet-152 (78.3%) and BiT-L (ResNet-based, 87.5% with ImageNet-21K pre-training).
• On CIFAR-10/100 trained from scratch, ViT massively overfits and underperforms simple CNNs.
• The model has the capacity to learn but lacks the structural constraints that help small-data learning.

Pre-training at scale provides ViT with the diverse examples needed to learn inductive biases from data:

Data scaling results (ViT-B/16 fine-tuned on ImageNet):

Pre-training Dataset	Images	ViT-B/16 Top-1	BiT-L (ResNet) Top-1
ImageNet-1K	1.2M	77.9%	76.8%
ImageNet-21K	14M	83.6%	84.7%
JFT-300M	303M	88.5%	87.5%

The crossover point: At ~30M pre-training images, ViT and BiT (ResNet) perform similarly. Below 30M, BiT wins. Above 30M, ViT pulls ahead and the gap widens with more data.

ViT model configurations and their results on ImageNet (pre-trained on JFT-300M):

Model	Layers	Heads	Hidden $D$	MLP $D$	Params	Top-1
ViT-B/16	12	12	768	3072	86M	84.2%
ViT-L/16	24	16	1024	4096	307M	87.8%
ViT-H/14	32	16	1280	5120	632M	88.6%

Why large-scale pre-training works for ViT:

1. Spatial relationships learned empirically: With 300M diverse images, the model sees enough examples to learn that adjacent patches are usually related, even without a locality bias.
2. Translation invariance from augmentation: Heavy data augmentation (random crops, flips) plus massive data diversity ensures the model sees patterns at all positions.
3. Multi-scale features emerge: Analysis of attention patterns shows that lower layers develop local attention (texture-level) while upper layers develop global attention (object-level) -- the hierarchy emerges without being architecturally imposed.

Computational economics (the key practical argument):

• ViT-L/16 pre-trained on JFT-300M requires ~2,500 TPUv3 core-days.
• BiT-L (ResNet-152x4) pre-trained on JFT-300M requires ~9,900 TPUv3 core-days.
• ViT achieves better accuracy with ~4x less compute at the largest scale. The Transformer's efficiency advantage grows with model size.

Transfer learning protocol:

1. Pre-train on large dataset (JFT-300M or ImageNet-21K) at $224 \times 224$
2. Remove pre-training classification head
3. Add new linear head for target task
4. Fine-tune at higher resolution ($384 \times 384$ or $512 \times 512$) with interpolated position embeddings
5. Use lower learning rate for fine-tuning (typically $10^{-4}$ to $3 \times 10^{-5}$)

Original ViT had a critical practical limitation:

• JFT-300M is proprietary to Google and not publicly available. Without it, ViT-B/16 achieves only 77.9% on ImageNet -- below ResNets.
• Training cost: Even if the data were available, pre-training ViT-H/14 on 300M images requires 2,500 TPUv3 core-days (~$50K+ in cloud compute).
• Research accessibility: Most academic labs have access to ImageNet-1K (1.2M images) and limited compute budgets. ViT's results were impressive but unreproducible for most of the community.
• The fundamental question: Is ViT's advantage inherent to the Transformer architecture, or is it just about having more data than others? Can we make Transformers work with publicly available data?

Three landmark follow-up papers solved the data efficiency problem from different angles:

DeiT (Data-efficient Image Transformer, Touvron et al., 2021):

Training ViT effectively on ImageNet-1K alone through better training recipes:

• Knowledge distillation: A pre-trained CNN (RegNetY-16GF) serves as a "teacher." A special distillation token (like [CLS]) is added to the sequence, trained to match the teacher's output via hard-label distillation.
• Heavy augmentation: RandAugment, MixUp ($\alpha = 0.8$), CutMix ($\alpha = 1.0$), random erasing, color jitter.
• Regularization: Stochastic depth (drop path rate 0.1), repeated augmentation, label smoothing ($\epsilon = 0.1$).
• Results: DeiT-B achieves 83.1% top-1 on ImageNet-1K with NO external data. DeiT-B distilled reaches 84.2% -- approaching ViT-B/16 pre-trained on ImageNet-21K (83.6%).

BEiT (BERT Pre-training for Image Transformers, Bao et al., 2022):

Self-supervised pre-training via masked image modeling:

• Tokenization: A discrete VAE (dVAE) tokenizes each image into a grid of visual tokens (vocabulary size 8192).
• Masking: Randomly mask 40% of patches in the input.
• Prediction: The Transformer predicts the dVAE token ID for each masked patch (classification over 8192-class vocabulary).
• Key insight: Predicting discrete tokens is more stable than predicting raw pixels because it captures semantic structure rather than low-level statistics.
• Results: BEiT-L achieves 88.6% top-1 after self-supervised pre-training on ImageNet-21K + fine-tuning on ImageNet-1K.

MAE (Masked Autoencoders, He et al., 2022):

Extremely efficient self-supervised pre-training through aggressive masking:

• Masking ratio: 75% of patches are masked -- far more aggressive than BERT's 15% or BEiT's 40%.
• Asymmetric encoder-decoder: The encoder only processes visible patches (25% of 196 = 49 tokens). A lightweight decoder reconstructs all patches including masked ones.
• Reconstruction target: Raw pixel values (MSE loss), not discrete tokens.
• Why 75% works: Images have enormous spatial redundancy. With 75% masking, the remaining patches do not form a trivially interpolatable pattern, forcing the model to learn deep semantic understanding.
• Efficiency: Since the encoder only processes 49 tokens (not 196), pre-training is ~3x faster and uses ~3x less memory than standard ViT training.
• Results: MAE with ViT-H achieves 87.8% top-1 on ImageNet-1K, using only ImageNet-1K for both pre-training and fine-tuning.

Impact on the field: These variants collectively demonstrated that ViT's advantage is architectural, not just data-driven. DeiT showed supervised training can work with the right recipe. BEiT and MAE showed self-supervised pre-training can replace massive labeled datasets entirely.

Understanding learned representations is critical for:

• Debugging failures: When ViT misclassifies an image, attention maps can reveal whether it focused on the wrong region (e.g., background instead of object) or the right region with wrong features.
• Trust and deployment: Medical imaging, autonomous driving, and other safety-critical applications require model interpretability. Attention maps provide direct evidence of what the model "looks at."
• Scientific understanding: Do Transformers learn the same features as CNNs (edges, textures, shapes)? Do they develop novel representations?
• Architecture design: Attention patterns reveal redundancy across heads and layers, informing efficient architecture design (pruning, distillation).

The challenge: ViT-B/16 has 12 layers with 12 attention heads each, producing $12 \times 12 = 144$ attention matrices of shape $197 \times 197$. How do we extract a meaningful summary from this high-dimensional representation?

Attention visualization methods and findings:

1. Raw Attention Maps (per head, per layer): Each attention head produces a $197 \times 197$ matrix. To visualize what a specific head attends to, extract the [CLS] token's attention row (shape $197$), discard position 0, and reshape the remaining 196 values to $14 \times 14$. Bilinearly upsample to the original image resolution.

2. Attention Rollout (Abnar & Zuidema, 2020): To capture cumulative information flow across layers, multiply attention matrices: $\tilde{A} = \prod_{l=1}^{L} A_l$

where $A_l$ is the attention matrix at layer $l$ (averaged across heads). Since attention is not a simple linear chain (residual connections add identity), a more accurate version is: $\hat{A}_l = 0.5 \cdot A_l + 0.5 \cdot I$

This accounts for the skip connection that adds the identity mapping at each layer.

Key findings from attention analysis:

Layer-wise progression:

• Layers 1-3 (early): Attention patterns resemble local filters -- heads attend to nearby patches, capturing texture and edge information. Some heads show stripe patterns (horizontal/vertical edge detectors).
• Layers 4-8 (middle): Attention becomes more semantic. Heads begin attending to object parts: a head might attend to all "face" patches or all "sky" patches, regardless of spatial distance.
• Layers 9-12 (late): [CLS] token attention becomes highly focused on discriminative regions. For a dog image, attention concentrates on the face and body while ignoring background.

Head specialization:

• Some heads in early layers act as "horizontal line detectors" (attending to patches in the same row)
• Others act as "color patch groupers" (attending to patches with similar color)
• Late heads develop "object part" attention (attending to semantically related patches)

Emergent segmentation: The [CLS] attention map from the last layer often closely matches object segmentation masks, even though ViT was trained only for classification -- no segmentation labels were ever provided. This led to methods like DINO (Caron et al., 2021) that extract high-quality segmentation from self-supervised ViT features.