“This Kernel Was Faster Yesterday” — In Pursuit of High-Fidelity GPU Kernel Benchmarking

“This Kernel Was Faster Yesterday” — In Pursuit of High-Fidelity GPU Kernel Benchmarking

“This Kernel Was Faster Yesterday” — In Pursuit of High-Fidelity GPU Kernel Benchmarking ← Blog 24 February 2026 “This Kernel Was Faster Yesterday” — In Pursuit of High-Fidelity GPU Kernel Benchmarking After Salvador Dalí’s The Persistence of Memory TL;DR GPU timing is deceptively hard and highly variable : power limits, thermal state, clock behavior, idle transitions, caching, and measurement methods all matter. High-fidelity evaluation is critical, especially for automated RL systems. In high-value matmul kernels, where even 5% matters, measurement noise can look like real gains and mislead

related reading

• Strangely, Matrix Multiplications on GPUs Run Faster When Given "Predictable" Data! [short]thonking.ai
• How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklogsiboehm.com
• Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com
• KernelBench v0.1 | Scaling Intelligence Lab at Stanford Universityscalingintelligence.stanford.edu
• The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
• KernelBench: Can LLMs Write GPU Kernels?scalingintelligence.stanford.edu
• GitHub - adam-maj/tiny-gpu: A minimal GPU design in Verilog to learn how GPUs work from the ground up · GitHubgithub.com
• Modern GPU Programming For MLSys — Modern GPU Programming For MLSysmlc.ai
• The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
• GitHub - wafer-ai/gpu-perf-engineering-resources: A curated resource list for learning AI performance engineering, from GPU fundamentals to production inference.github.com
• How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu