Compiling Models to Megakernels - by Luminal and Joe Fioti

Fine-grained synchronization, deep pipelines, and zero kernel launch overheads, automatically.

Compiling Models to Megakernels Fine-grained synchronization, deep pipelines, and zero kernel launch overheads, automatically. Luminal and Joe Fioti Jan 09, 2026 8 2 Share Luminal is an inference compiler, and as such we’re interested in driving inference right up to the physical limits of the hardware. Inference has two fundamental limitations: compute (flops) and bandwidth (TB/s). Increasing these two requires buying much more expensive hardware, so we want to make sure we’re using all the compute and bandwidth we have available to us! This basically boils down to: anytime the GPU is not loa

saved by

• Alex K

related reading

• Mirage Persistent Kernel: A Compiler and Runtime for Mega-Kernelizing Tensor Programsarxiv.org
• Real-time LLM Inference on Standard Datacenter GPUs (3,000 tokens/s per request)blog.kog.ai
• Kimi-K2.5 Inference Benchmark - Luminalluminal.com
• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
• How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklogsiboehm.com
• Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Making Deep Learning go Brrrr From First Principleshorace.io
• Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• How To Scale Your Modeljax-ml.github.io
• [2410.20399] ThunderKittens: Simple, Fast, and Adorable AI Kernelsarxiv.org