Compiling Models to Megakernels - by Luminal and Joe Fioti

Fine-grained synchronization, deep pipelines, and zero kernel launch overheads, automatically.

Compiling Models to Megakernels Fine-grained synchronization, deep pipelines, and zero kernel launch overheads, automatically. Luminal and Joe Fioti Jan 09, 2026 8 2 Share Luminal is an inference compiler, and as such we’re interested in driving inference right up to the physical limits of the hardware. Inference has two fundamental limitations: compute (flops) and bandwidth (TB/s). Increasing these two requires buying much more expensive hardware, so we want to make sure we’re using all the compute and bandwidth we have available to us! This basically boils down to: anytime the GPU is not loa

saved by

• Alex K

related reading

• Mirage Persistent Kernel: A Compiler and Runtime for Mega-Kernelizing Tensor Programsarxiv.org
• Real-time LLM Inference on Standard Datacenter GPUs (3,000 tokens/s per request)blog.kog.ai
• Kimi-K2.5 Inference Benchmark - Luminalluminal.com
• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
• How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklogsiboehm.com
• Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com
• Making Deep Learning go Brrrr From First Principleshorace.io
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Modern GPU Programming For MLSys — Modern GPU Programming For MLSysmlc.ai
• Mixture-of-Kittens: our open-source MoE megakernel for NVL72scursor.com
• GitHub - adam-maj/tiny-gpu: A minimal GPU design in Verilog to learn how GPUs work from the ground up · GitHubgithub.com
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev