v9.1.1
Menu
Paper breakdownEfficient AttentionPRO2022

FlashAttention

Fast and Memory-Efficient Exact Attention with IO-Awareness

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré

7

Concepts

Paper Overview

FlashAttention (Dao, Fu, Ermon, Rudra & Ré — Stanford and SUNY Buffalo, NeurIPS 2022) is best understood as a correction to what the field had been optimizing. Self-attention is quadratic in sequence length, so for five years the response was to make it cheaper in FLOPs — sparse approximations, low-rank approximations, and combinations of the two. Many of these reduced compute to linear or near-linear in NN. Many of them were also, on real hardware, no faster.

The paper's diagnosis is that FLOPs were the wrong currency. On an A100, on-chip SRAM runs at roughly 1919 TB/s while HBM runs at 1.51.5–2.02.0 TB/s, and compute throughput has outpaced memory throughput for years. Most operations in a Transformer are therefore bottlenecked by memory accesses, not arithmetic. An algorithm that cuts FLOPs while still writing an N×NN \times N matrix to HBM has optimized the term that was not binding.

So FlashAttention changes the objective to IO-awareness: count the reads and writes between HBM and SRAM, and minimize those. The mechanism is tiling — process attention in blocks small enough to live in SRAM — which runs immediately into the obstacle that softmax couples every column in a row, so the normalizer is not known until the whole row is seen. The resolution is an online softmax that carries two running statistics and rescales as it goes, producing the exact same output as standard attention rather than an approximation of it.

The payoff is unusual in that nothing is traded away. Attention computation itself runs 7.6x faster, memory footprint drops from quadratic to linear (up to 20x less), and the backward pass gets faster despite doing more FLOPs. BERT-large trains 15% faster than the MLPerf 1.1 record; GPT-2 up to 3x faster than HuggingFace. And because 64K-token sequences now fit in memory, block-sparse FlashAttention produced the first sequence model to beat chance on Path-256.

Explainer video
Preview · first 30s

Chapter Roadmap

Click any topic to jump in

1
The GPU Memory Hierarchy

SRAM is ~10x faster than HBM and ~4000x smaller. Compute has outpaced memory, so attention is bottlenecked by data movement, not arithmetic.

measured against
2
What Standard Attention Costs

Three kernels, each round-tripping an N×N matrix through HBM. The FLOPs are fine; the traffic is the problem.

restructured by
3
Tiling and the Online Softmax

Softmax couples the whole row, so it resists tiling. Carrying two running statistics (m, ℓ) makes it decomposable — exactly, not approximately.

made trainable by
4
Recomputation and Kernel Fusion

Throw away the N×N matrices and rebuild them in SRAM during the backward pass. More FLOPs, less traffic, net faster.

analyzed as

Exact bound and approximate extension

5
IO Complexity and Optimality

Θ(N²d²M⁻¹) HBM accesses versus Θ(Nd + N²), with a matching lower bound proving no exact algorithm does asymptotically better.

6
Block-Sparse FlashAttention

Skip the masked-out blocks entirely and the IO cost scales directly with the sparsity — the one part of the paper that is approximate.

validated by
7
Results

15% over the MLPerf BERT record, 3x on GPT-2, 20x less attention memory — and the first Transformer to beat chance on Path-X.

Concept by concept

7 sections, in reading order.

PRO

Unlock the full breakdown

All 7 concepts — figures, key points, formulas and interactive demos — are part of PixelBank Premium.

  1. 01The GPU Memory Hierarchy
  2. 02What Standard Attention Actually Costs
  3. 03Tiling and the Online Softmax
  4. 04Recomputation and Kernel Fusion
  5. 05IO Complexity and Optimality
  6. 06Block-Sparse FlashAttention
  7. 07Results