Chapter 8: Advanced Kernels & Optimization

You now have the full toolkit — the hardware model, the Triton programming model, fusion, reductions, tiling, and tuning. This final chapter applies it to the kernels that show up in every transformer, and points you toward what's next.

Fused layer normalization combines the softmax-style row reduction (mean, variance) with element-wise scale-and-shift in a single pass — the same fusion principle, applied again. Low-memory dropout shows a beautiful trick: instead of storing a giant random mask, regenerate it on the fly from a seed, trading a little compute for a lot of memory. Fused attention (FlashAttention) is the capstone idea: it combines tiling, the online-softmax running-max/running-sum trick from Chapter 6, and fusion to compute attention without ever materializing the $O(N^2)$ score matrix — the optimization that made long-context transformers practical. Persistent kernels keep programs resident and loop over work to amortize launch overhead.

The meta-message: you won't memorize these kernels — you'll recognize the principles in them. Every one is keep-data-on-chip, reuse-it, fuse-the-chain, hide-the-latency, tune-empirically. That's the durable skill this course was building toward.

This chapter covers:

• Fused Layer Normalization: reduction + scale/shift in one pass
• Low-Memory Dropout: regenerate the mask from a seed
• Fused Attention (FlashAttention): attention without the N×N matrix
• Persistent Kernels: amortizing launch overhead
• Where to Go Next: extending your Triton skills