Self-Supervised2022

Masked Autoencoders

Self-Supervised Visual Pre-training

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick

Paper Overview

Masked Autoencoders (MAE) demonstrate that a simple self-supervised approach — masking 75% of image patches and learning to reconstruct them — produces powerful visual representations that scale to very large models.

The key insight is an asymmetric encoder-decoder design: a heavy Vision Transformer encoder processes only the visible 25% of patches, while a lightweight decoder reconstructs the full image from encoded visible patches plus learned mask tokens. This asymmetry makes pre-training 3-4× faster than processing all patches.

Key advances:

75% masking ratio: Much higher than BERT's 15% — images have more spatial redundancy than language
Asymmetric architecture: Heavy encoder on 25% of tokens, light decoder on all tokens — 3× speedup
Scalable pre-training: Enables ViT-Huge (632M params) to achieve 86.9% on ImageNet
Simple pixel reconstruction: MSE on normalized pixels — no tokenizer, no contrastive pairs needed
Strong transfer: +2 AP on COCO detection, +3.7 mIoU on ADE20K segmentation vs. supervised

MAE showed that self-supervised pre-training for vision can be as simple and scalable as BERT is for language, reigniting interest in masked image modeling and paving the way for foundation models in computer vision.

Chapter Roadmap

Click any topic to jump in

75% Masking

Aggressive masking creating a non-trivial reconstruction task.

Random Masking

Uniform per-patch random masking breaks spatial correlations.

complements

Asymmetric Arch

Heavy encoder on visible patches, light decoder on full set.

Pixel Reconstruction

Per-patch normalized MSE as the simple, effective objective.

enables

Scaling Behavior

Accuracy keeps improving up to ViT-Huge unlike supervised baselines.

Transfer Learning

Strong downstream fine-tuning across detection, segmentation, classification.

MAE uses a heavy encoder that only processes visible patches and a lightweight decoder that reconstructs all patches, achieving a 3-4× training speedup over processing all tokens.

The Problem

The computational cost of BERT-style masking for vision:

BERT masks 15% of tokens and replaces them with [MASK] tokens, but still processes all tokens (including masks) through the full Transformer encoder. For vision:

ViT-L splits a 224×224 image into 196 patches (14×14 grid of 16×16 patches)
With 75% masking, 147 patches are masked and 49 are visible
A BERT-style approach would process all 196 tokens through the heavy encoder
But 147 of those tokens are just [MASK] — carrying no actual image information

The efficiency waste:

Processing mask tokens through the encoder is pure waste:

Mask tokens contain no image content — just a shared learned embedding
The encoder's job is to build rich representations from image data
Processing 147 identical mask tokens adds 3× more computation for no extra signal
ViT-L encoder has 24 blocks — that is 24 × 147 = 3,528 wasted token-block interactions

Why this matters:

Self-supervised pre-training requires many epochs (MAE uses 1,600). Even a 3× speedup means training in days instead of weeks, or enabling experiments on larger models.

The Solution

Asymmetric design: Separate the heavy encoding from the lightweight reconstruction.

Encoder (heavy, 25% tokens):

Only the visible patches (25%) enter the encoder:

Linear patch embedding: each 16×16 patch → $\mathbf{e}_i \in \mathbb{R}^d$
Add positional embeddings (only for visible positions)
Process through the full ViT encoder (e.g., ViT-L: 24 blocks, 1024-d, 16 heads)

$\mathbf{z}_i = \text{ViT}(\mathbf{e}_i + \mathbf{p}_i), \quad i \in \text{visible}$

The encoder never sees mask tokens. With 75% masking, it processes only 49 out of 196 tokens — a 4× reduction in encoder computation.

Decoder (lightweight, all tokens):

After encoding, combine visible encoded tokens with learned mask tokens:

Insert shared learned mask token $\mathbf{m}$ at masked positions
Add positional embeddings to all tokens (visible + mask)
Process through a small decoder (e.g., 8 blocks, 512-d)
Linear projection to predict pixel values for each patch

$\hat{\mathbf{x}} = \text{Decoder}([\mathbf{z}_{\text{visible}}, \mathbf{m}_{\text{masked}}] + \mathbf{p}_{\text{all}})$

Decoder design:

The decoder is intentionally small — about 10% of the encoder's computation:

8 Transformer blocks (vs. 24 in encoder)
512-d width (vs. 1024 in encoder)
Processes all 196 tokens but is much lighter per token

Why asymmetry works:

The encoder builds rich representations from visible patches. The decoder's job is simpler: use those representations plus positional context to fill in the gaps. This simpler task doesn't need a heavy model.

Speed comparison:

Approach	Tokens in Encoder	Relative Training Time
Full (all tokens)	196	1.0×
MAE (25% visible)	49	~0.27×

Key Points

Encoder processes only 25% of patches — 4× fewer tokens than full processing

Decoder is ~10% of encoder compute: 8 blocks vs. 24, 512-d vs. 1024-d

Mask tokens are shared learned embeddings — never enter the encoder

Positional embeddings added before decoder so mask tokens know their position

Net training speedup: ~3-4× compared to encoding all tokens

Mathematical Formulation

Encoder (visible patches only)

$\mathbf{z} = \text{ViT}_{\text{encoder}}(\{\mathbf{e}_i + \mathbf{p}_i\}_{i \in \text{visible}})$

Only visible patch embeddings (with their positional encodings) enter the heavy ViT encoder. Masked positions are completely absent from this computation.

Decoder (all tokens)

$\hat{\mathbf{x}} = \text{Decoder}([\mathbf{z}_{\text{visible}}, \mathbf{m}_{\text{masked}}] + \mathbf{p}_{\text{all}})$

The decoder receives encoded visible tokens plus shared mask tokens at masked positions. Positional embeddings are added so the decoder knows where each token belongs.

Mathematical Intuition

The encoder processes only visible patches ( $25\%$ of tokens), so its cost drops from $O((HW)^2)$ to $O((0.25 HW)^2) \approx 6\%$ of the full cost. The decoder is shallow (8 layers) and reconstructs all patches. Asymmetry means $\sim 3\times$ wall-clock speedup on the same hardware while keeping the heavy lifting in the encoder for transfer.

Premium Content

Upgrade to PixelBank Premium to unlock this content.

Back to All Papers

Study Plans

Computer Vision Study Plan All Papers Practice Problems

Masked Autoencoders

Paper Overview

Chapter Roadmap

75% Masking

Random Masking

Asymmetric Arch

Pixel Reconstruction

Scaling Behavior

Transfer Learning

Asymmetric Encoder-Decoder

The Problem

The Solution

Key Points

Mathematical Formulation

Encoder (visible patches only)

Decoder (all tokens)

High Masking Ratio

Random Masking Strategy

Reconstruction Target

Scaling Properties

Transfer Learning

Premium Content

Related Papers

Study Plans