Masked Autoencoders

The computational cost of BERT-style masking for vision:

BERT masks 15% of tokens and replaces them with [MASK] tokens, but still processes all tokens (including masks) through the full Transformer encoder. For vision:

• ViT-L splits a 224×224 image into 196 patches (14×14 grid of 16×16 patches)
• With 75% masking, 147 patches are masked and 49 are visible
• A BERT-style approach would process all 196 tokens through the heavy encoder
• But 147 of those tokens are just [MASK] — carrying no actual image information

The efficiency waste:

Processing mask tokens through the encoder is pure waste:

• Mask tokens contain no image content — just a shared learned embedding
• The encoder's job is to build rich representations from image data
• Processing 147 identical mask tokens adds 3× more computation for no extra signal
• ViT-L encoder has 24 blocks — that is 24 × 147 = 3,528 wasted token-block interactions

Why this matters:

Self-supervised pre-training requires many epochs (MAE uses 1,600). Even a 3× speedup means training in days instead of weeks, or enabling experiments on larger models.

Asymmetric design: Separate the heavy encoding from the lightweight reconstruction.

Encoder (heavy, 25% tokens):

Only the visible patches (25%) enter the encoder:

1. Linear patch embedding: each 16×16 patch → $\mathbf{e}_i \in \mathbb{R}^d$
2. Add positional embeddings (only for visible positions)
3. Process through the full ViT encoder (e.g., ViT-L: 24 blocks, 1024-d, 16 heads)

$\mathbf{z}_i = \text{ViT}(\mathbf{e}_i + \mathbf{p}_i), \quad i \in \text{visible}$

The encoder never sees mask tokens. With 75% masking, it processes only 49 out of 196 tokens — a 4× reduction in encoder computation.

Decoder (lightweight, all tokens):

After encoding, combine visible encoded tokens with learned mask tokens:

1. Insert shared learned mask token $\mathbf{m}$ at masked positions
2. Add positional embeddings to all tokens (visible + mask)
3. Process through a small decoder (e.g., 8 blocks, 512-d)
4. Linear projection to predict pixel values for each patch

$\hat{\mathbf{x}} = \text{Decoder}([\mathbf{z}_{\text{visible}}, \mathbf{m}_{\text{masked}}] + \mathbf{p}_{\text{all}})$

Decoder design:

The decoder is intentionally small — about 10% of the encoder's computation:

• 8 Transformer blocks (vs. 24 in encoder)
• 512-d width (vs. 1024 in encoder)
• Processes all 196 tokens but is much lighter per token

Why asymmetry works:

The encoder builds rich representations from visible patches. The decoder's job is simpler: use those representations plus positional context to fill in the gaps. This simpler task doesn't need a heavy model.

Speed comparison: