GPU Basics with Triton Study Plan
An 8-chapter curriculum that takes you from "what is a warp?" to writing fused, tiled GPU kernels in Triton that rival hand-tuned CUDA — all in Python.
Recommended Study Path
Prerequisites
Foundations Study Plan
- Python Foundations
- NumPy & Arrays
- Basic Linear Algebra
Complete the Foundations study plan first →
GPU Fundamentals
Weeks 1-2
- Ch 1: Why GPUs?
- Ch 2: Architecture & Memory
Warps, SMs, the memory hierarchy
Writing Kernels
Weeks 2-5
- Ch 3-4: Triton & Vector Add
- Ch 5: Autotuning & Benchmarks
program_id, offsets, masking, BLOCK_SIZE
High Performance
Weeks 5-8
- Ch 6-7: Fusion & Matmul
- Ch 8: Advanced Kernels
Fused softmax, tiled matmul, attention
All Chapters
Why GPUs? The Parallel Computing Mindset
Why GPUs exist, CPU vs GPU design philosophy, latency vs throughput, the SIMT execution model, when parallel hardware wins, and Amdahl's law.
GPU Architecture & the Memory Hierarchy
Threads, warps, blocks and streaming multiprocessors, the register→SRAM→DRAM hierarchy, memory coalescing, bandwidth vs latency, and the golden rule of keeping data high in the hierarchy.
Introduction to Triton
What Triton is and why it exists, Triton vs CUDA vs PyTorch, the block-oriented programming model, programs, grids and tiles, and the @triton.jit decorator.
Your First Kernel: Vector Addition
Building the canonical Triton kernel step by step: program ids, computing offsets, loading and storing with pointers, masking the ragged tail, choosing BLOCK_SIZE, and launching the grid.
Memory Access, Autotuning & Benchmarking
Writing memory-bound kernels that hit peak bandwidth: coalesced access patterns, autotuning over block sizes and num_warps, benchmarking with triton.testing, and reading a roofline.
Kernel Fusion: Fused Softmax
Why fusion is the central optimization on GPUs: counting DRAM round-trips, the row-wise reduction pattern, numerically stable softmax, keeping a row in SRAM, and the fused-vs-unfused speedup.
Matrix Multiplication
The kernel that powers deep learning: blocked/tiled matmul, accumulating in SRAM, the inner-k loop, super-grouping program ids for L2 cache reuse, and approaching cuBLAS performance.
Advanced Kernels & Optimization
Putting it together: fused layer normalization, low-memory dropout with seeded RNG, the ideas behind fused attention (FlashAttention), persistent kernels, and where to go next.
Practice Problem Sets
Sharpen your skills with coding challenges and system design problems.
Curriculum designed to take you from GPU fundamentals to writing high-performance Triton kernels that approach cuBLAS-level speed.