✳flâneur — a map of the web's best reading

Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordić

aleksagordic.com · 13,656 words · saved by 9 readers

In this post, I will gradually introduce all of the core hardware concepts and programming techniques that underpin state-of-the-art (SOTA) NVIDIA GPU matrix-multiplication (matmul) kernels. Why matmul? Transformers spend most of their FLOPs inside matmuls (linear layers in MLP, attention QKV projections, output projections, etc.) both during training and inference. These operations are embarrassingly parallel, making them a natural fit for GPUs. Finally, understanding how matmul kernels work gives you the toolkit to design nearly any other high-performance GPU kernel. This post is structured into four parts: My aim is for this post to be self-contained: detailed enough to stand on its own, yet concise enough to avoid becoming a textbook. This is the first part of a broader series. In the following posts, I (aspirationally) plan to cover: To write performant GPU kernels, you need a solid mental model of the hardware. This will become clear very quickly as we dive into hardware architec

In this post, I will gradually introduce all of the core hardware concepts and programming techniques that underpin state-of-the-art (SOTA) NVIDIA GPU matrix-multiplication (matmul) kernels. Why matmul? Transformers spend most of their FLOPs inside matmuls (linear layers in MLP, attention QKV projections, output projections, etc.) both during training and inference. These operations are embarrassingly parallel, making them a natural fit for GPUs. Finally, understanding how matmul kernels work gives you the toolkit to design nearly any other high-performance GPU kernel. This post is structured

Explore this link on the map →

saved by

related reading