A Tiny Compiler for Data-Parallel Kernels — Andrew Healey
healeycodes.com · 1,290 words · saved by 1 readers
Exploring how compilers lower ordinary loops into explicit data-parallel kernels.
A lot of fast code starts as a boring loop. Modern hardware can perform the same operation on multiple values at once (e.g. SIMD and SIMT), and sometimes we write code directly for those execution models but other times, a compiler starts with regular-looking code and rewrites it so multiple loop iterations can run together. I built a tiny compiler (~180LOC of Python) to understand what that transformation looks like. My compiler lowers kernels (rewrites them into a simpler, more explicit form where data parallelism is visible). The input is a small hand-written AST, and the output is a…
saved by
related reading
- Modern GPU Programming For MLSys — Modern GPU Programming For MLSysmlc.ai
- How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklogsiboehm.com
- A friendly introduction to machine learning compilers and optimizershuyenchip.com
- GitHub - adam-maj/tiny-gpu: A minimal GPU design in Verilog to learn how GPUs work from the ground up · GitHubgithub.com
- Lecture 117:Building a GPU Compiler from Scratchyoutube.com
- Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com
- Tiny TPUtinytpu.com
- Compiling Models to Megakernels - by Luminal and Joe Fiotiblog.luminal.com
- Writing a C compiler in 500 lines of Pythonvgel.me
- Implementing a GPU's Programming Model on a CPUlitherum.blogspot.com
- LLVM is Smarter Than Meblog.sulami.xyz
- Horace He: Building Machine Learning Systems for a Trillion Trillion Floating Point Operationsyoutube.com