KernelBench: Can LLMs Write GPU Kernels? | Scaling Intelligence Lab at Stanford University

A benchmark designed to evaluate the ability of LLMs to generate efficient GPU kernels for optimizing neural network performance We introduce KernelBench, a benchmark designed to evaluate the ability of large language models (LLMs) to generate efficient GPU kernels for optimizing neural network performance. With 250 well-defined neural network tasks spanning foundational operators, simple fusion patterns, and full ML architectures, the benchmark tasks LLMs to replace PyTorch implementations with custom kernels that are correct and performant. KernelBench highlights the potential for agentic optimization for computer systems with dense feedback signal, where systems iteratively refine kernel designs using profiling tools and tight feedback loops to achieve near-peak hardware utilization. As models scale, well-optimized kernels have far-reaching implications, from reducing the massive energy demands of AI systems to enabling fair and efficient comparisons of novel architectures. By provi

A benchmark designed to evaluate the ability of LLMs to generate efficient GPU kernels for optimizing neural network performance TL;DR We introduce KernelBench, a benchmark designed to evaluate the ability of large language models (LLMs) to generate efficient GPU kernels for optimizing neural network performance. With 250 well-defined neural network tasks spanning foundational operators, simple fusion patterns, and full ML architectures, the benchmark tasks LLMs to replace PyTorch implementations with custom kernels that are correct and performant. KernelBench highlights the potential for…

saved by

• abinaya dinesh
• Ratan Kaliani
• Ishaan Panigrahi

related reading

• Modern GPU Programming For MLSys — Modern GPU Programming For MLSysmlc.ai
• LLM Engineer's Almanac - Advisormodal.com
• Recursive Language Models — Alex Zhang, MIT PhDyoutube.com
• How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklogsiboehm.com
• GitHub - wafer-ai/gpu-perf-engineering-resources: A curated resource list for learning AI performance engineering, from GPU fundamentals to production inference.github.com
• PostTrainBenchposttrainbench.com
• KernelBench v0.1 | Scaling Intelligence Lab at Stanford Universityscalingintelligence.stanford.edu
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
• Kevin-32B: Multi-Turn RL for Writing CUDA Kernels | Cognitioncognition.ai
• [2410.20399] ThunderKittens: Simple, Fast, and Adorable AI Kernelsarxiv.org
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com