flâneur

Retire the Abstractions · Hazy Research

hazyresearch.stanford.edu · 1,046 words · saved by 3 readers

Last year, we set out to write megakernels. But implementing megakernels can be quite miserable. It requires complex data structures, extensive inter-thread/SM/GPU synchronization, deeply nested control flow, etc. We couldn't hold it in our heads, so we did what computer science has done for 70 years: we built a layer of abstraction. Even with it, we had to fight through race conditions and deadlocks for a couple months before the Llamas ran blazingly fast. This year, we built an MoE megakernel, but we deleted the abstraction. With agents, we could work through the complexity directly and build target-optimized code from scratch. No intermediate layers of C++ abstraction were needed. So what could this mean? We've noticed an interesting pattern over the past quarter. The tasks we could already do without abstractions (e.g., writing an optimized GEMM kernel) are now nearly automated given the right prompt. For now, we still have to tell the agent which PTX instructions to use, the warp-

TL;DR: We think CUDA DSLs are heading toward retirement Last year, we set out to write megakernels. But implementing megakernels can be quite miserable. It requires complex data structures, extensive inter-thread/SM/GPU synchronization, deeply nested control flow, etc. We couldn't hold it in our heads, so we did what computer science has done for 70 years: we built a layer of abstraction. Even with it, we had to fight through race conditions and deadlocks for a couple months before the Llamas ran blazingly fast. This year, we built an MoE megakernel, but we deleted the abstraction. With…

saved by

related reading