Chapter 3: Parallel Patterns: Reductions, Atomics & Tiling
The three patterns behind most real GPU kernels: combining many values into one with a shared-memory tree reduction, letting threads update a shared location safely with atomics, and reusing data through shared-memory tiles in matrix multiplication.
Chapter Overview
One-thread-per-element kernels are the easy case: every thread writes its own output and nobody steps on anyone else. Real workloads are harder because threads must combine their results — summing an array, building a histogram, multiplying matrices — and the moment two threads want to update the same location, you have a coordination problem.
This final chapter covers the three patterns that solve it and appear in almost every serious CUDA kernel:
- The Reduction Problem: why combining values is fundamentally different from mapping them
- Tree Reduction in Shared Memory: collapsing N values to one in log N parallel steps
- Atomics: race-free read-modify-write on a shared location, and the price of contention
- Tiled Matrix Multiplication: shared-memory tiles that turn a memory-bound matmul into a fast one
Together these turn the basics from the first two chapters into kernels that actually compete with library code.
Chapter Roadmap
Click any topic to jump in
The Reduction Problem
Combining many values into one races when threads share a target; associativity enables a parallel tree.
Tree Reduction in Shared Memory
Halve the stride each step with a barrier between rounds: N values to one in log N parallel steps.
Atomics
Indivisible read-modify-write avoids lost updates; contention serializes, so reduce-then-atomic.
Tiled Matrix Multiplication
Shared-memory tiles read each value once per tile instead of once per output, raising arithmetic intensity.
Sign up to unlock this chapter
This chapter is part of PixelBank Premium. Create a free account, then upgrade to read the full lesson — concepts, walkthroughs, and exercises.