FlashAttention
Fast and Memory-Efficient Exact Attention with IO-Awareness
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré
7
Concepts
Paper Overview
FlashAttention (Dao, Fu, Ermon, Rudra & Ré — Stanford and SUNY Buffalo, NeurIPS 2022) is best understood as a correction to what the field had been optimizing. Self-attention is quadratic in sequence length, so for five years the response was to make it cheaper in FLOPs — sparse approximations, low-rank approximations, and combinations of the two. Many of these reduced compute to linear or near-linear in . Many of them were also, on real hardware, no faster.
The paper's diagnosis is that FLOPs were the wrong currency. On an A100, on-chip SRAM runs at roughly TB/s while HBM runs at – TB/s, and compute throughput has outpaced memory throughput for years. Most operations in a Transformer are therefore bottlenecked by memory accesses, not arithmetic. An algorithm that cuts FLOPs while still writing an matrix to HBM has optimized the term that was not binding.
So FlashAttention changes the objective to IO-awareness: count the reads and writes between HBM and SRAM, and minimize those. The mechanism is tiling — process attention in blocks small enough to live in SRAM — which runs immediately into the obstacle that softmax couples every column in a row, so the normalizer is not known until the whole row is seen. The resolution is an online softmax that carries two running statistics and rescales as it goes, producing the exact same output as standard attention rather than an approximation of it.
The payoff is unusual in that nothing is traded away. Attention computation itself runs 7.6x faster, memory footprint drops from quadratic to linear (up to 20x less), and the backward pass gets faster despite doing more FLOPs. BERT-large trains 15% faster than the MLPerf 1.1 record; GPT-2 up to 3x faster than HuggingFace. And because 64K-token sequences now fit in memory, block-sparse FlashAttention produced the first sequence model to beat chance on Path-256.
Chapter Roadmap
Click any topic to jump in
The GPU Memory Hierarchy
SRAM is ~10x faster than HBM and ~4000x smaller. Compute has outpaced memory, so attention is bottlenecked by data movement, not arithmetic.
What Standard Attention Costs
Three kernels, each round-tripping an N×N matrix through HBM. The FLOPs are fine; the traffic is the problem.
Tiling and the Online Softmax
Softmax couples the whole row, so it resists tiling. Carrying two running statistics (m, ℓ) makes it decomposable — exactly, not approximately.
Recomputation and Kernel Fusion
Throw away the N×N matrices and rebuild them in SRAM during the backward pass. More FLOPs, less traffic, net faster.
Exact bound and approximate extension
IO Complexity and Optimality
Θ(N²d²M⁻¹) HBM accesses versus Θ(Nd + N²), with a matching lower bound proving no exact algorithm does asymptotically better.
Block-Sparse FlashAttention
Skip the masked-out blocks entirely and the IO cost scales directly with the sparsity — the one part of the paper that is approximate.
Results
15% over the MLPerf BERT record, 3x on GPT-2, 20x less attention memory — and the first Transformer to beat chance on Path-X.
Concept by concept
7 sections, in reading order.
Unlock the full breakdown
All 7 concepts — figures, key points, formulas and interactive demos — are part of PixelBank Premium.
- 01The GPU Memory Hierarchy
- 02What Standard Attention Actually Costs
- 03Tiling and the Online Softmax
- 04Recomputation and Kernel Fusion
- 05IO Complexity and Optimality
- 06Block-Sparse FlashAttention
- 07Results
Keep reading
Related Papers
Go deeper