Chapter 2: GPU Architecture & the Memory Hierarchy
Map the physical GPU: how threads group into warps and blocks, how Streaming Multiprocessors schedule them, the register→shared→L2→global memory hierarchy, why coalesced memory access is critical, and how arithmetic intensity and the roofline decide whether you are compute- or memory-bound.
- Topics
- 5
- Demo
- 1
- Videos
- 3
- Min read
- 16
Chapter Overview
Chapter 1 gave you the mindset; this chapter gives you the map. To write fast kernels you must know where your data lives and how the hardware moves it.
A GPU is organized as a grid of Streaming Multiprocessors (SMs), each containing many ALUs, a register file, an on-chip scratchpad (shared memory / L1), and warp schedulers. Your kernel launches a grid of blocks; the GPU assigns blocks to SMs, and each SM runs the block's threads as warps. Threads in a block can cooperate through fast shared memory; threads in different blocks essentially cannot.
The single most important structure here is the memory hierarchy. Registers are nearly free but tiny and private. Shared memory (SRAM) is fast and block-local — and in Triton it's the resource you orchestrate every time you 'tile' a computation. L2 is a device-wide cache. Global memory (DRAM) is huge but slow, and its bandwidth is the bottleneck for most kernels. Almost every optimization in this course reduces to: read from DRAM as few times as possible, do as much work as you can while the data is on-chip.
This chapter covers:
- Threads, Warps & Blocks: the execution hierarchy
- Streaming Multiprocessors: where blocks actually run
- The Memory Hierarchy: registers → SRAM → L2 → DRAM
- Memory Coalescing: turning many small reads into one fast transaction
- Arithmetic Intensity & the Roofline: are you compute- or memory-bound?
Chapter Roadmap
Click any topic to jump in
Threads, Warps & Blocks
The execution hierarchy: warps of 32 run in lockstep; blocks cooperate via shared memory; blocks are independent.
Streaming Multiprocessors
Where blocks run — registers, schedulers, ALUs, and SRAM. Resource limits set occupancy.
The Memory Hierarchy
Registers → SRAM → L2 → DRAM. Keep hot data high and reuse it before it falls back down.
Memory Coalescing
Contiguous warp reads become one transaction; strided reads waste most of the bandwidth.
Arithmetic Intensity & Roofline
FLOPs/byte decides whether you're memory- or compute-bound — and therefore what to optimize.
Topics
5 topics in this chapter
Chapter 1 argued that a GPU needs tens of thousands of threads in flight to hide memory latency. That raises an organizational problem: how do you name, schedule and coordinate that many threads without the programmer tracking each one? GPUs answer with a fixed three-level hierarchy. Threads, the unit you write code for, are bundled into warps of 32 that share one instruction stream, the SIMT model from the previous chapter. Warps are grouped into blocks, which the hardware places on a single streaming multiprocessor so their threads can share fast on-chip memory and wait at barriers. Blocks form the grid, which is everything one kernel launch does. Each level has its own rules about what threads may share and when they may synchronize, and confusing them produces kernels that are either slow or silently wrong. This topic walks the hierarchy with concrete counts: a grid of 1,024 blocks of 256 threads is 262,144 threads in 8,192 warps. It then shows why blocks must be independent of each other, and how Triton's program instance maps onto the block level. The next topic opens up the SM itself.
Definition
The GPU execution hierarchy has four levels. A thread is one scalar instance of the kernel; a warp is 32 threads issued together; a block (cooperative thread array) is a group of warps resident on one SM that can share memory and synchronize at barriers; and a grid is the set of independent blocks launched by one kernel call.
The Execution Hierarchy
Threads are the smallest unit; each has its own registers and program counter. Warps are fixed groups of 32 threads that execute in lockstep (the SIMT unit from Chapter 1). Blocks (a.k.a. thread blocks or CTAs) are groups of warps that run together on a single SM and can cooperate via shared memory and barriers (__syncthreads). The grid is all the blocks for one kernel launch.
Key rules: threads in the same block can share data and synchronize; threads in different blocks cannot (blocks may run in any order, even on different SMs, possibly not at the same time). This independence is what lets the GPU scale — it can schedule blocks however many SMs it has.
In Triton you mostly think one level up: a program corresponds to a block-sized tile of work, and Triton manages the threads within it for you.
An SM has hard limits: max resident threads, warps, blocks, registers, and shared memory. Occupancy = (resident warps) / (max warps). If each block uses registers/thread and bytes of shared memory, the SM can host blocks. Choosing block size and resource usage is really choosing occupancy.
You launch a kernel with a grid of 1024 blocks, each with 256 threads. How many threads and warps is that, and what can block #0's threads do that block #5's cannot?
Related Problems on PixelBank
The previous topic said blocks are placed onto streaming multiprocessors, but treated the SM as a black box. Opening it explains the resource limits that decide how fast nearly every kernel runs. An SM is a self-contained processor: a register file divided among its resident threads, warp schedulers that pick a ready warp each cycle, FP32 and integer lanes, tensor cores on recent parts, load/store units, and a block of on-chip SRAM split between shared memory and L1 cache. A GPU is essentially many copies of this unit sharing an L2 cache and DRAM; the RTX 2060 used throughout this chapter has 30 SMs, while an H100 SXM has 132. Because every resident thread needs its own registers and every resident block needs its shared memory, an SM can hold only as many warps as its scarcest resource allows. That count, relative to the hardware maximum, is occupancy, and it determines how much latency the schedulers can hide. This topic works through the arithmetic: on an SM with a 65,536-register file, a kernel that uses 128 registers per thread fits only 512 threads. The next topic follows the data those threads load.
Definition
A streaming multiprocessor (SM) is the GPU's replicated processing unit. It holds a register file, warp schedulers, arithmetic lanes, tensor cores on recent architectures, and configurable shared memory and L1 cache. Blocks are assigned to SMs whole, and the number that can be resident at once is limited by registers, shared memory, and warp and block slots, which together set occupancy.
Inside an SM
Each Streaming Multiprocessor contains: a large register file split among resident threads, CUDA cores (FP32/INT ALUs) and often Tensor Cores for matrix math, several warp schedulers that issue instructions, an on-chip shared memory / L1 cache, and load/store units. A high-end GPU has dozens to over a hundred SMs.
The GPU's block scheduler hands blocks to SMs until their resources (registers, shared memory, warp slots) are full. The warp schedulers on each SM then pick a ready warp every cycle and issue its next instruction — switching warps for free when one stalls (latency hiding). More resident warps = more options to hide stalls = higher utilization.
Resident threads where is registers/thread and is shared memory/block. The binding constraint sets occupancy. High occupancy isn't always optimal (a few big-tile warps can beat many small ones for compute-bound kernels), but very low occupancy almost always means exposed latency.
Your kernel uses 128 registers per thread. The SM has a 65,536-register file and can hold at most 2,048 threads. How many threads can actually be resident, and what does that do to occupancy?
Theory Exercise
Problem:
Two kernels do the same total work. Kernel A launches 64 blocks of 1024 threads; Kernel B launches 4096 blocks of 16 threads. On a GPU with 80 SMs, which is likely to use the hardware better, and why?
Hints:
- A warp is 32 threads — what happens with a 16-thread block?
- Think about how many blocks each SM can hold and warp utilization
- Consider whether each SM gets enough warps to hide latency
Related Problems on PixelBank
The SM from the previous topic can issue a fused multiply-add on every lane every cycle, but only if operands arrive fast enough. They usually do not. An RTX 2060 can perform about 6.5 trillion FP32 operations per second, yet its DRAM delivers 336 GB/s, roughly 84 billion floats. Each float fetched from DRAM must therefore feed about 77 operations just to keep the arithmetic busy. No single memory can be both large and fast, so GPUs stack several. Registers are private to a thread and effectively free to read. Shared memory and L1 are on-chip SRAM, tens of kilobytes per SM, with latencies of roughly 20 to 35 cycles (a T4 L1 hit measured 32). L2 is a few megabytes shared by all SMs: 3 MB on the RTX 2060, 50 MB on an H100. DRAM holds gigabytes but costs hundreds of cycles per miss; microbenchmarks on a Turing T4 measured 188 cycles for an L2 hit and 296 for an L2 miss. This topic tours each level with real sizes and latencies and shows how tiling turns repeated DRAM reads into on-chip reuse, the idea behind fast matmul and FlashAttention.
Definition
The GPU memory hierarchy orders storage from fastest and smallest to slowest and largest: per-thread registers, per-SM shared memory and L1 cache (on-chip SRAM), a device-wide L2 cache, and off-chip global memory (DRAM or HBM). Performance depends on staging data into the upper levels once and reusing it many times before it is evicted.
Registers, SRAM, L2, and DRAM
From fastest and smallest to slowest and largest:
- Registers: private to a thread, read at full speed. 256 KB per SM (65,536 32-bit registers).
- Shared memory / L1 (SRAM): about 20-35 cycles, shared by a block. 64 KB per SM on Turing, up to 228 KB on Hopper. Stage tiles here.
- L2 cache: about 200 cycles, shared by all SMs. 3 MB on an RTX 2060, 50 MB on an H100.
- Global memory (DRAM/HBM): about 300-600 cycles, gigabytes in size, high bandwidth but high latency.
The rule: move data up once and reuse it many times before it falls back down.
Think in traffic, not latency: latency is hidden by other warps, but bytes moved are not. If a tile of values is loaded once into SRAM and each value is reused times, DRAM traffic falls from to values, a factor of . A kernel needs roughly FLOPs per byte to stay compute-bound: on an RTX 2060, about 77 FLOPs per 4-byte float. Only reuse from SRAM and registers can supply that.
A naive matmul reads each element of A and B from global memory once per output it contributes to. A tiled matmul loads a block of A and B into shared memory once and reuses it for many outputs. Why is the tiled version dramatically faster?
The memory hierarchy topic treated a DRAM access as fetching one value, but the hardware never moves a single float. Global memory is read in fixed-size chunks: the L1 and L2 caches track 32-byte sectors, and DRAM itself transfers bursts of similar size. When the 32 threads of a warp issue a load, the memory system gathers their addresses and fetches every sector they touch. If the 32 addresses are consecutive 4-byte floats, that is 128 bytes in four sectors, and every byte fetched is used. If each thread reads an address far from its neighbors, every lane drags in a whole sector to use 4 bytes of it, and most of the bandwidth is thrown away. This is memory coalescing, often the biggest difference between a naive kernel and a fast one. This topic measures it on a real GPU. A Triton kernel reading floats with stride 1 reached 303 GB/s on an RTX 2060; the same kernel at stride 2 reached 156 GB/s, and from stride 16 upward it settled at 20 GB/s, 15 times slower. You will also see why tl.arange over the contiguous axis coalesces by construction.
Definition
Memory coalescing is the combining of a warp's simultaneous global-memory accesses into the minimum number of 32-byte sector transactions. An access pattern is coalesced when consecutive lanes touch consecutive addresses, so every fetched byte is used; it is uncoalesced when lanes are spread across many sectors, wasting most of the fetched bandwidth.
Coalesced vs Strided Access
Global memory moves in 32-byte sectors. When a warp issues a load, the hardware fetches every sector its 32 lane addresses touch. If lane reads (contiguous floats), 32 lanes need 4 sectors and every byte is used: coalesced. If lane reads for a large stride , each lane lands in its own sector and uses 4 of 32 or more bytes: uncoalesced.
So layout matters: walking the contiguous (last) axis of a row-major tensor is coalesced; walking down a column is not. In Triton, offsets built with tl.arange over the contiguous dimension stay coalesced by construction.
Useful bandwidth . A 32-byte-sector model predicts slowdown for float32 stride . Measured on an RTX 2060: 1.94x, 3.82x and 7.57x at strides 2, 4 and 8, matching it, but then 15x from stride 16 (64 bytes) upward, so DRAM fetched about 64 bytes per scattered float on this GDDR6 card. Older material quotes 32x from 128-byte lines; real loss depends on the part.
You process a row-major matrix M[row][col]. One kernel has each warp read down a column (M[0][c], M[1][c], …); another reads along a row (M[r][0], M[r][1], …). Which is coalesced?
Related Problems on PixelBank
The last three topics gave two separate ceilings: the arithmetic rate of the SMs and the bandwidth of the memory system, made worse by uncoalesced access. Every kernel hits one of them first, and optimizing against the wrong one wastes effort. Vectorizing the math of a kernel that is starved for data does nothing; neither does fusing loads in a kernel whose ALUs are already saturated. Arithmetic intensity, the number of floating-point operations performed per byte moved from DRAM, decides which ceiling applies. Element-wise float32 addition does one FLOP per 12 bytes, an intensity of 0.083. A large matrix multiply reuses each loaded value many times, and its intensity grows with tile size. The roofline model of Williams, Waterman and Patterson turns this into a picture: attainable performance is the lesser of peak compute and intensity times peak bandwidth, and the two lines meet at the ridge point, about 19 FLOPs per byte on an RTX 2060. This topic places real kernels on that plot. Our 50-million-element add reached 89 percent of peak bandwidth yet 0.4 percent of peak FLOP/s, and that is the best it can do. Later chapters use the roofline to justify fusion and tiled matmul.
Definition
Arithmetic intensity is the ratio of floating-point operations to bytes moved between DRAM and the chip. The roofline model bounds attainable throughput by ; kernels with below the ridge point are memory-bound, and those above it are compute-bound, limited by arithmetic throughput.
FLOPs per Byte
Arithmetic intensity is FLOPs divided by bytes moved from DRAM. The roofline model plots attainable performance against : a slanted line (memory-bound, performance ) meeting a flat ceiling (compute-bound, peak FLOP/s) at the ridge point.
Low- kernels (float32 element-wise ops, about 0.08 to 0.25 FLOP/byte) sit on the slope: memory-bound, so the fix is fewer DRAM bytes (fusion, better layout), not faster math. High- kernels (large matmuls, where grows with tile size) sit under the ceiling: compute-bound, so the fix is better use of ALUs and tensor cores.
The ridge point sits at . A kernel with can never exceed FLOP/s no matter how good your math is — performance is bandwidth intensity. Fusion raises by cutting the denominator (bytes), sliding the kernel up the slanted roof toward the ridge.
Vector add c = a + b on float32 arrays does 1 add per output but reads 8 bytes (a, b) and writes 4 bytes per output. What is its arithmetic intensity, and is it compute- or memory-bound?