Worklog: Optimising GEMM on NVIDIA H100 for cuBLAS-like Performance (WIP) | Hamza's Blog
Matrix multiplication sits at the core of modern deep learning. Whether it is transformers, CNNs, or even simple MLPs, everything eventually reduces to G...
Worklog: Optimising GEMM on NVIDIA H100 for cuBLAS-like Performance (WIP) – Hamza's Blog Worklog: Optimising GEMM on NVIDIA H100 for cuBLAS-like Performance (WIP) 12 Jan, 2026 🚧 Work in progress. Please reach out on LinkedIn if you spot any mistakes. Introduction Matrix multiplication sits at the core of modern deep learning. Whether it is transformers, CNNs, or even simple MLPs, everything eventually reduces to GEMM. GPUs are built to run this operation at scale, and libraries like cuBLAS set the performance bar with kernels tuned down to the last instruction. In this blog I am rebuilding th
saved by
related reading
- How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklogsiboehm.com
- CVPR2023_eff_tutorial_molchanov.pdfnvlabs.github.io
- Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com
- NVIDIA Tensor Core Evolution: From Volta To Blackwellnewsletter.semianalysis.com
- GPU Performance Background User's Guide - NVIDIA Docsdocs.nvidia.com
- Python API reference — nvMatmulHeuristicsdocs.nvidia.com
- How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
- Outperforming cuBLAS on H100: a Worklogcudaforfun.substack.com
- Outperforming cuBLAS on H100: a Worklogcudaforfun.substack.com
- Mini Project: GPU Accelerated Matrix Multiplication (almost) like cuBLAS0mean1sigma.com
- CUTLASS: Fast Linear Algebra in CUDA C++ | NVIDIA Technical Blogdeveloper.nvidia.com
- How to Think About TPUs | How To Scale Your Modeljax-ml.github.io