flâneur

Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Research

hazyresearch.stanford.edu · 2,590 words · saved by 9 readers

Benjamin Spector*, Jordan Juravsky*, Stuart Sul, Owen Dugan, Dylan Lim, Dan Fu, Simran Arora, Chris Ré There are some applications that benefit from running LLMs really, really fast. This low-latency regime encompasses applications like chatbots and human-in-the-loop workflows, where users care a lot about seeing responses come back immediately. Given the importance of these low-latency workloads, we wanted to explore just how fast we can run open-source models on modern GPUs. To really stress-test existing systems, we consider an aggressive low-latency scenario where we generate a single sequence with Llama-3.2-1B. This workload is strongly memory bound – our performance is dominated by how fast we can load model weights from GPU global memory. It turns out that popular LLM inference engines – vLLM and SGLang – are only able to use at most 50% of available GPU bandwidth when running this workload on an H100. The root of the problem, which we'll describe more below, is that existing sy

Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Research May 27, 2025 · 13 min read Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B Benjamin Spector* , Jordan Juravsky* , Stuart Sul , Owen Dugan , Dylan Lim , Dan Fu , Simran Arora , Chris Ré There are some applications that benefit from running LLMs really, really fast. This low-latency regime encompasses applications like chatbots and human-in-the-loop workflows, where users care a lot about seeing responses come back immediately. Given the importance of these low-latency workloads, we wanted t

saved by

related reading