A Tiny Compiler for Data-Parallel Kernels — Andrew Healey
healeycodes.com · 1,290 words · saved by 1 readers
Exploring how compilers lower ordinary loops into explicit data-parallel kernels.
A lot of fast code starts as a boring loop. Modern hardware can perform the same operation on multiple values at once (e.g. SIMD and SIMT), and sometimes we write code directly for those execution models but other times, a compiler starts with regular-looking code and rewrites it so multiple loop iterations can run together. I built a tiny compiler (~180LOC of Python) to understand what that transformation looks like. My compiler lowers kernels (rewrites them into a simpler, more explicit form where data parallelism is visible). The input is a small hand-written AST, and the output is a…
saved by
related reading
- Modern GPU Programming For MLSys — Modern GPU Programming For MLSysmlc.ai
- A friendly introduction to machine learning compilers and optimizershuyenchip.com
- How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklogsiboehm.com
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com
- LLVM is Smarter Than Meblog.sulami.xyz
- Mirage Persistent Kernel: A Compiler and Runtime for Mega-Kernelizing Tensor Programsarxiv.org
- KernelBench: Can LLMs Write GPU Kernels?scalingintelligence.stanford.edu
- Compiling Models to Megakernels - by Luminal and Joe Fiotiblog.luminal.com
- Building a Performant Compiler from Scratch — Bowen Chengbowencheng.dev
- GitHub - adam-maj/tiny-gpu: A minimal GPU design in Verilog to learn how GPUs work from the ground up · GitHubgithub.com
- siboehmsiboehm.com