Worklog: Optimising GEMM on NVIDIA H100 for cuBLAS-like Performance (WIP) | Hamza's Blog
Matrix multiplication sits at the core of modern deep learning. Whether it is transformers, CNNs, or even simple MLPs, everything eventually reduces to G...
Worklog: Optimising GEMM on NVIDIA H100 for cuBLAS-like Performance (WIP) – Hamza's Blog Worklog: Optimising GEMM on NVIDIA H100 for cuBLAS-like Performance (WIP) 12 Jan, 2026 🚧 Work in progress. Please reach out on LinkedIn if you spot any mistakes. Introduction Matrix multiplication sits at the core of modern deep learning. Whether it is transformers, CNNs, or even simple MLPs, everything eventually reduces to GEMM. GPUs are built to run this operation at scale, and libraries like cuBLAS set the performance bar with kernels tuned down to the last instruction. In this blog I am rebuilding th
saved by
related reading
- How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklogsiboehm.com
- Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com
- CVPR2023_eff_tutorial_molchanov.pdfnvlabs.github.io
- NVIDIA Tensor Core Evolution: From Volta To Blackwellnewsletter.semianalysis.com
- H100 vs GB200 NVL72 Training Benchmarks – Power, TCO, and Reliability Analysis, Software Improvement Over Time – SemiAnalysissemianalysis.com
- How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
- Outperforming cuBLAS on H100: a Worklogcudaforfun.substack.com
- GPU Performance Background User's Guide - NVIDIA Docsdocs.nvidia.com
- Outperforming cuBLAS on H100: a Worklogcudaforfun.substack.com
- Grouped GEMM for Imbalanced Experts on Blackwell: A WIP Workloggauravjain.bearblog.dev
- Strangely, Matrix Multiplications on GPUs Run Faster When Given "Predictable" Data! [short]thonking.ai
- Implementing a fast Tensor Core matmul on the Ada Architecture | spatters.caspatters.ca