flâneur

Outperforming cuBLAS on H100: a Worklog

cudaforfun.substack.com · 5,523 words · saved by 2 readers

In this post, we’ll iteratively implement a CUDA kernel for matrix multiplication on latest generation1 NVIDIA hardware: H100. We’ll gain a deep understanding of H100 architecture and showcase these optimizations step by step. The final kernel outperforms cuBLAS by 7% for N=4096. It fits in a single C++ file without any dependencies. This post is intended as a sequel to Simon’s legendary blog which showcases similar optimizations for A6000 GPU. However H100 GPUs are completely different beasts, requiring entirely different algorithms. As an example, algorithm from Simon’s blog is only able to achieve 4% of cuBLAS performance2. In this post, we will pick up from Simon’s blog and iteratively reach 107% of cuBLAS. All my code is available on Github. Thanks for reading Pranjal’s Substack! Subscribe for free to receive new posts and support my work. Our aim is not to be a cuBLAS replacement, but to design a slightly faster, yet simplistic, matmul kernel which works for generally large matri

Outperforming cuBLAS on H100: a Worklog CUDA matmul kernel - from scratch Pranjal Shankhdhar Nov 29, 2024 131 6 9 Share In this post, we’ll iteratively implement a CUDA kernel for matrix multiplication on latest generation 1 NVIDIA hardware: H100. We’ll gain a deep understanding of H100 architecture and showcase these optimizations step by step. The final kernel outperforms cuBLAS by 7% for N=4096 . It fits in a single C++ file without any dependencies. This post is intended as a sequel to Simon’s legendary blog which showcases similar optimizations for A6000 GPU. However H100 GPUs are complet

saved by

related reading