Chapter 1: Why GPUs? The Parallel Computing Mindset
Understand why GPUs exist, how their design philosophy differs from CPUs, the difference between latency and throughput, the SIMT execution model, when parallel hardware actually helps, and the hard ceiling Amdahl's law puts on speedup.
- Topics
- 5
- Videos
- 5
- Min read
- 17
Chapter Overview
Before you write a single line of Triton, you need to think like the hardware. GPUs are not 'fast CPUs' — they are a fundamentally different kind of processor built around one idea: do an enormous number of simple operations at the same time.
A modern CPU has a handful of powerful cores, each loaded with branch predictors, deep caches, and out-of-order execution machinery to make a single thread finish as fast as possible. A GPU spends that same silicon budget on thousands of simpler cores. It doesn't try to make any one operation fast — it tries to keep all of those cores busy so that the total amount of work per second is staggering.
This chapter builds the mental model that everything else depends on: the distinction between latency (how long one task takes) and throughput (how much work finishes per second), the SIMT execution model where threads move in lockstep groups called warps, the kinds of problems where this hardware shines, and Amdahl's law, which tells you that the serial part of your program — not the parallel part — ultimately limits your speedup.
This chapter covers:
- CPUs vs GPUs: two different answers to 'how do we make computers fast?'
- Latency vs Throughput: the single most important trade-off in this course
- The SIMT Execution Model: warps, lockstep, and divergence
- When GPUs Win (and Lose): matching the problem to the hardware
- Amdahl's Law: the ceiling on every speedup you will ever measure
Chapter Roadmap
Click any topic to jump in
CPUs vs GPUs
Two design philosophies: few powerful cores (latency) vs thousands of simple cores (throughput).
Latency vs Throughput
GPUs hide latency with massive parallelism instead of reducing it — occupancy keeps the ALUs fed.
SIMT Execution Model
Threads run in lockstep warps of 32; data-dependent branches cause divergence and serialize work.
When GPUs Win (and Lose)
Data-parallel, high-arithmetic-intensity work wins; serial, branchy, or tiny work belongs on the CPU.
Amdahl's Law
The serial fraction caps speedup at 1/(1-p) — optimization means shrinking the serial part.
Topics
5 topics in this chapter
Suppose you must apply the same arithmetic to fifty million numbers, and two chips sit on your desk. One finishes any single operation in a few nanoseconds but works on only a few dozen values at once; the other is slower per operation but keeps tens of thousands of operations in flight. Which one finishes first depends on the workload, not on which chip is "faster", and that question is the mindset this whole plan builds on. This first topic compares the two design philosophies. A CPU spends its transistor budget on making one instruction stream fast: deep caches, branch prediction and out-of-order execution. A GPU spends it on arithmetic lanes, memory bandwidth and the scheduling hardware that keeps them busy. We will measure what that means on a real machine, where a 50-million-element add runs about 13 times faster on an RTX 2060 than on an 8-core Ryzen, yet becomes almost 6 times slower once the PCIe copies are counted. The topics that follow explain why: latency versus throughput, the SIMT execution model, the shapes of problems GPUs win on, and Amdahl's ceiling on any speedup.
Definition
A CPU is a latency-optimized processor: a few powerful cores with large caches, branch prediction and out-of-order execution, built to finish one instruction stream quickly. A GPU is a throughput-optimized processor: many simple arithmetic lanes grouped into streaming multiprocessors, with little control logic per lane, built to maximize the total operations completed per second across many independent threads.
Two Design Philosophies
A CPU spends most of its transistors on control and cache: branch predictors, out-of-order schedulers and multi-megabyte caches that make one instruction stream run fast. It has a handful to around a hundred very capable cores, plus 8-wide AVX2 SIMD units.
A GPU spends most of its transistors on arithmetic: thousands of simple lanes grouped into Streaming Multiprocessors, with small caches shared by many threads. Each lane is simpler than a CPU core, but there are thousands, fed by far more bandwidth (336 GB/s on an RTX 2060 versus about 51 GB/s for dual-channel DDR4-3200). CPUs win on sequential, branchy work; GPUs win when one operation applies to massive data.
Peak arithmetic throughput is roughly . An RTX 2060 has 1,920 FP32 lanes at a 1.68 GHz boost clock doing one fused multiply-add (2 FLOPs) per cycle: about 6.5 TFLOP/s. An 8-core Zen 3 CPU has two 8-wide FMA units per core, 32 FLOPs per cycle per core, so about 1 TFLOP/s near 4 GHz. Measured on a 4096 x 4096 float32 matmul, the CPU reached 0.84 TFLOP/s and the GPU 6.3 (7.5x); with FP16 tensor cores the GPU reached 28.8 TFLOP/s (34x). Lane count, not clock speed, dominates.
You add two float32 arrays of 50,000,000 elements, c[i] = a[i] + b[i]. With NumPy on an 8-core Ryzen 7 5800X the add takes 26.5 ms; on an RTX 2060 the kernel takes 2.0 ms. Why is the GPU faster, and why does the comparison flip if the arrays start in CPU memory?
Related Problems on PixelBank
The previous topic said a GPU trades fast individual operations for many simultaneous ones. That trade pays off only if something covers the slow parts, and on a GPU the slowest part is memory: a load that misses the caches takes hundreds of cycles to return. Microbenchmarks of a Turing T4 measured 296 cycles for an L2 miss and 616 when the TLB misses too. A core that simply waited would sit idle almost all the time. Two numbers describe any processor here. Latency is the time one operation takes from start to finish; throughput is how many operations finish per second. CPUs attack latency directly with large caches and speculation. GPUs mostly accept it and attack throughput instead, keeping so many independent warps resident that some are always ready to issue while others wait. This topic makes that precise. You will see how warp switching hides stalls, why Little's law tells you how much parallel work is needed to cover a given latency, and why occupancy, the share of warp slots actually filled, is the first number to check when a kernel underperforms. The next topic looks at what a warp actually is.
Definition
Latency is the time from issuing one operation to its result being available; throughput is the number of operations completed per unit time. Latency hiding is the GPU strategy of keeping many independent warps resident on each SM so that, while some wait on memory, others issue instructions, keeping the arithmetic units busy without making any single access faster.
Hiding Latency with Parallelism
A GPU load that misses the caches takes hundreds of cycles; a CPU's DRAM miss is just as slow, but the CPU hides it with deep caches, prefetching and out-of-order execution. GPU lanes have none of that machinery, so the GPU hides latency through parallelism instead: when one warp stalls on memory, the scheduler issues from another warp that is ready, at no switching cost. With enough warps in flight, the arithmetic units never go idle.
This is why GPUs want many more threads than lanes: oversubscription gives the scheduler a deep pool of work. The metric is occupancy: the share of warp slots resident and available to hide latency.
Little's Law makes this precise: to fully hide a memory latency (cycles) you need enough in-flight work to cover it. If each warp issues a memory request every cycles, you need roughly warps resident to keep the pipeline full. With and , that's ~25 warps per SM — which is exactly why high occupancy matters.
A single GPU thread reads from global memory (≈500 cycles latency) then does one multiply. Most of its life is spent waiting. How does the GPU still hit near-peak throughput?
Theory Exercise
Problem:
Your kernel achieves only 30% of the GPU's peak throughput. A profiler shows the ALUs are idle most of the time, waiting on memory, and occupancy is very low (only 2 warps per SM resident). Explain what is happening in terms of latency hiding, and what you would change.
Hints:
- Idle ALUs + waiting on memory means stalls are not being hidden
- Latency hiding requires many resident warps to switch between
- What limits how many warps can be resident? (registers, shared memory, block size)
Related Problems on PixelBank
Latency hiding needs thousands of threads, but a chip cannot afford a separate instruction fetch and decode unit for each one. GPUs resolve this by grouping threads. On NVIDIA hardware 32 threads form a warp, and the scheduler issues one instruction for the whole warp, which its 32 lanes execute on their own data. This is SIMT, single instruction, multiple threads: you write code for one thread, and the hardware runs it 32 at a time. Building on the previous topic, the warp is also the unit the scheduler swaps in and out to hide memory latency. The design is cheap in silicon, but it has one sharp edge. When lanes of the same warp disagree at a branch, the warp cannot execute both sides at once, so it runs each side in turn with the other lanes masked off. We will measure that cost directly: a kernel whose branch split every warp in half ran exactly twice as slowly as the same kernel with warp-aligned branches. You will also see why Triton, which works on whole tiles with masks instead of per-thread branches, avoids most of this problem by construction.
Definition
SIMT (single instruction, multiple threads) is the GPU execution model in which threads are grouped into warps, 32 threads on NVIDIA hardware. The scheduler issues one instruction per warp and each active lane executes it on its own registers. When lanes take different branch paths, the warp executes each path serially with the inactive lanes masked, which is called branch divergence.
Warps and Lockstep
Threads are grouped into warps of 32 on NVIDIA hardware. The scheduler issues one instruction per warp and 32 lanes execute it on different data, so one control unit drives 32 ALUs. Since Volta each thread has its own program counter, but the hardware still issues to a warp's active lanes together.
The catch is branch divergence. If a conditional sends some lanes down the if path and others down the else path, the warp runs the if body with the other lanes masked off, then the else body. Both paths run serially, so a split warp pays for both bodies, which can halve throughput or worse.
If a branch splits a warp's active lanes into a nonempty group taking path A (cost ) and a nonempty group taking path B (cost ), the warp's time is whatever the split, not the average you might hope for. Only when all 32 lanes agree ( or ) does the warp pay for a single path. A switch whose cases are all taken inside one warp costs the sum of paths, up to 32 serial passes.
Inside a kernel you write if (idx % 2 == 0) heavy_path() else light_path(). Threads in the same warp take different branches. What happens to performance?
Related Problems on PixelBank
With CPUs, latency hiding and warps in hand, the practical question becomes which programs actually belong on a GPU. Porting code costs real engineering time, and a surprising amount of ported code ends up slower. The trouble is that "GPUs are fast" hides several conditions that must all hold at once. The work must split into many independent pieces; the pieces should follow the same control path so warps do not diverge; memory accesses should be regular; and there must be enough work to amortize fixed costs. Those costs are real. On our test machine a tiny kernel launch costs about 5 microseconds of host time, eight times the 0.64 microseconds NumPy needs for an 8-element add, and moving 200 MB across PCIe takes about 15 milliseconds. This topic turns the conditions into a checklist. You will sort real workloads, from a 50-million-element ReLU to a linked-list walk, into GPU and CPU piles, and meet arithmetic intensity, the FLOPs-per-byte ratio that the roofline model in Chapter 2 develops fully. The final topic of the chapter then asks how much speedup is possible even when the GPU fits.
Definition
A workload suits a GPU when it is data-parallel (many independent elements receive the same operation), has regular control flow and memory access, is large enough to amortize kernel-launch and host-to-device transfer costs, and needs enough arithmetic or memory bandwidth per element to exploit the GPU's large advantage in compute throughput and DRAM bandwidth over a CPU.
Data Parallelism and Arithmetic Intensity
GPUs win when a problem is data-parallel: the same operation applied independently to many data elements, with few data-dependent branches and predictable memory access. Element-wise ops, matrix multiply, convolutions, and reductions all fit.
GPUs lose when work is inherently serial (each step depends on the previous), highly branchy (lots of divergence), latency-critical for a single item (you need one answer now, not a million answers soon), or tiny (the cost of launching a kernel and copying data to the GPU dwarfs the work). They also lose when the problem is dominated by moving data rather than computing on it — unless you can keep that data on-chip.
A useful screen is arithmetic intensity . Problems with high (matmul: ) keep the ALUs fed and approach peak FLOP/s. Problems with low (element-wise add: ) are limited by memory bandwidth no matter how many cores you have. GPUs win biggest when is high and the work is parallel.
Which of these belong on a GPU? (a) Apply ReLU to a 50M-element tensor. (b) Parse a 2 KB JSON config. (c) Multiply two 4096×4096 matrices. (d) Walk a linked list summing nodes.
Theory Exercise
Problem:
A colleague wants to move a recursive tree-traversal algorithm (each node visited depends on results from its parent) to the GPU 'because GPUs are fast'. Explain whether this is a good idea and what property of the problem decides it.
Hints:
- Is the work data-parallel or does each step depend on a previous one?
- What does a serial dependency chain do to thousands of cores?
- Think about divergence in an irregular tree
Related Problems on PixelBank
The previous topic sorted workloads into GPU-friendly and GPU-hostile ones, but most real programs are a mix: a highly parallel core surrounded by serial setup, data transfers and a final reduction. How much does accelerating only the parallel part help? Intuition says a thousand-times-faster kernel should make the whole program roughly a thousand times faster. It does not. If 5 percent of the runtime stays serial, the program can never run more than 20 times faster, no matter how many cores you add, because that serial slice keeps its original length while everything else shrinks toward zero. Gene Amdahl made this argument in 1967, and it remains the most useful sanity check in performance work. This topic derives the speedup formula and works through the diminishing returns it predicts: with 95 percent parallel code, 100 workers give 16.8x, 1,000 give 19.6x, and all further workers together add less than 0.4x. It then contrasts Amdahl with Gustafson's observation that bigger machines usually run bigger problems. The practical lesson closes the chapter: after the first big win, profile again and attack whatever is now serial, such as host-device copies.
Definition
Amdahl's law bounds the speedup of a fixed-size problem when only a fraction of its original runtime is accelerated by a factor : . As grows without bound the speedup approaches , so the serial fraction sets a hard ceiling that no amount of added parallel hardware can lift.
The Speedup Ceiling
Let be the fraction of your program that can be parallelized and the number of parallel workers. The overall speedup is bounded: as , . The serial fraction becomes the hard limit.
The lesson: if 90% of your runtime is parallelizable (), even infinite cores give at most a 10× speedup. The remaining 10% serial work dominates. This is why real GPU optimization is as much about shrinking the serial portion (data transfers, host-side setup, sequential reductions) as it is about parallelizing the rest.
Differentiating shows that as grows — added cores help less and less. The serial term sits in the denominator unaffected by , fixing the asymptote at . Gustafson's law offers a more optimistic view for problems that grow with more compute, but Amdahl governs a fixed problem size.
A program spends 95% of its time in a parallelizable loop and 5% in serial setup. What is the best possible speedup on a GPU with thousands of cores?