Outperforming cuBLAS on H100: a Worklog

CUDA matmul kernel - from scratch

Outperforming cuBLAS on H100: a Worklog CUDA matmul kernel - from scratch Pranjal Shankhdhar Nov 29, 2024 131 6 9 Share In this post, we’ll iteratively implement a CUDA kernel for matrix multiplication on latest generation 1 NVIDIA hardware: H100. We’ll gain a deep understanding of H100 architecture and showcase these optimizations step by step. The final kernel outperforms cuBLAS by 7% for N=4096 . It fits in a single C++ file without any dependencies. This post is intended as a sequel to Simon’s legendary blog which showcases similar optimizations for A6000 GPU. However H100 GPUs are complet

saved by

• Daivik Dhar

related reading

• Outperforming cuBLAS on H100: a Worklogcudaforfun.substack.com
• How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklogsiboehm.com
• Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com
• Worklog: Optimising GEMM on NVIDIA H100 for cuBLAS-like Performance (WIP) – Hamza's Bloghamzaelshafie.bearblog.dev
• Reverse-Engineering cuBLASaccu.org
• Mini Project: GPU Accelerated Matrix Multiplication (almost) like cuBLAS0mean1sigma.com
• Implementing a fast Tensor Core matmul on the Ada Architecture | spatters.caspatters.ca
• GPUs Go Brrr · Hazy Researchhazyresearch.stanford.edu
• CUTLASS: Fast Linear Algebra in CUDA C++ | NVIDIA Technical Blogdeveloper.nvidia.com
• The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
• GitHub - wangzyon/NVIDIA_SGEMM_PRACTICE: Step-by-step optimization of CUDA SGEMM · GitHubgithub.com
• CVPR2023_eff_tutorial_molchanov.pdfnvlabs.github.io