Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Research
Benjamin Spector*, Jordan Juravsky*, Stuart Sul, Owen Dugan, Dylan Lim, Dan Fu, Simran Arora, Chris Ré There are some applications that benefit from running LLMs really, really fast. This low-latency regime encompasses applications like chatbots and human-in-the-loop workflows, where users care a lot about seeing responses come back immediately. Given the importance of these low-latency workloads, we wanted to explore just how fast we can run open-source models on modern GPUs. To really stress-test existing systems, we consider an aggressive low-latency scenario where we generate a single sequence with Llama-3.2-1B. This workload is strongly memory bound – our performance is dominated by how fast we can load model weights from GPU global memory. It turns out that popular LLM inference engines – vLLM and SGLang – are only able to use at most 50% of available GPU bandwidth when running this workload on an H100. The root of the problem, which we'll describe more below, is that existing sy
Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Research May 27, 2025 · 13 min read Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B Benjamin Spector* , Jordan Juravsky* , Stuart Sul , Owen Dugan , Dylan Lim , Dan Fu , Simran Arora , Chris Ré There are some applications that benefit from running LLMs really, really fast. This low-latency regime encompasses applications like chatbots and human-in-the-loop workflows, where users care a lot about seeing responses come back immediately. Given the importance of these low-latency workloads, we wanted t
Explore this link on the map →saved by
related reading
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Best practices to accelerate inference for large-scale production workloadstogether.ai
- Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
- Compiling Models to Megakernels - by Luminal and Joe Fiotiblog.luminal.com
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- How is LLaMa.cpp possible?finbarr.ca
- Kevin-32B: Multi-Turn RL for Writing CUDA Kernels | Cognitioncognition.ai
- Real-time LLM Inference on Standard Datacenter GPUs (3,000 tokens/s per request)blog.kog.ai
- How To Scale Your Modeljax-ml.github.io
- How to Land a Frontier Lab Jobvladfeinberg.com
- LLM Engineer's Almanac - Workloads | Modalmodal.com