BrrrViz
GPU programming is full of concepts that make sense once you see them. BrrrViz turns the hard parts into things you can watch and interact with. ACT 01 How the GPU works. From execution to optimization. ACT 02 How ML is built on the GPU. Transformers, inference, training at scale, multi-GPU. Get notified when ML Systems drops.
A Graphics Processing Unit (GPU) is thousands of highly-parallel, small processors organized into synchronized groups, all sharing a strict memory hierarchy. GPUs don't run one thread at a time, they launch thousands of parallel processes. When threads in a warp take different paths, the GPU serializes execution. Learn the causes and mitigations. Threads in a block must coordinate access to shared memory. Learn why races happen, how barriers order access, and what they cost. Shared memory is striped across 32 banks. When multiple threads in a warp hit the same bank with different addresses, ac
saved by
related reading
- Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com
- How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklogsiboehm.com
- How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
- GitHub - adam-maj/tiny-gpu: A minimal GPU design in Verilog to learn how GPUs work from the ground up · GitHubgithub.com
- AMD GPUs go brrr · Hazy Researchhazyresearch.stanford.edu
- On the GPU Memory Hierarchygytis.co
- Execution Model - SLING user documentationdoc.sling.si
- Making Deep Learning go Brrrr From First Principleshorace.io
- GPUs Go Brrr · Hazy Researchhazyresearch.stanford.edu
- How To Scale Your Modeljax-ml.github.io
- GPU Performance Background User's Guide - NVIDIA Docsdocs.nvidia.com
- How to Think About TPUs | How To Scale Your Modeljax-ml.github.io