How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog

In this post, I’ll iteratively optimize an implementation of matrix multiplication written in CUDA.My goal is not to build a cuBLAS replacement, but to deepl...

How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog Subscribe Si_Boehm Comments (16) December 2022 In this post, I’ll iteratively optimize an implementation of matrix multiplication written in CUDA. My goal is not to build a cuBLAS replacement, but to deeply understand the most important performance characteristics of the GPUs that are used for modern deep learning. This includes coalescing global memory accesses, shared memory caching and occupancy optimizations, among others. You can download the code for all kernels from Github . Also checkout wangzyon’s repo from wh

saved by

• Amir
• Pranav
• Kevin Wang
• Varun Shenoy
• Ratan Kaliani
• Julian Quevedo
• Adithya V
• Hamidah Oderinwale
• Atem Aguer
• Benedict Neo
• Raayan Dhar
• Aaron Pham

related reading

• Outperforming cuBLAS on H100: a Worklogcudaforfun.substack.com
• Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com
• Worklog: Optimising GEMM on NVIDIA H100 for cuBLAS-like Performance (WIP) – Hamza's Bloghamzaelshafie.bearblog.dev
• Implementing a fast Tensor Core matmul on the Ada Architecture | spatters.caspatters.ca
• Outperforming cuBLAS on H100: a Worklogcudaforfun.substack.com
• GitHub - wangzyon/NVIDIA_SGEMM_PRACTICE: Step-by-step optimization of CUDA SGEMMgithub.com
• Strangely, Matrix Multiplications on GPUs Run Faster When Given "Predictable" Data! [short]thonking.ai
• CUTLASS: Fast Linear Algebra in CUDA C++ | NVIDIA Technical Blogdeveloper.nvidia.com
• Reverse-Engineering cuBLASaccu.org
• Learning CUDA by optimizing matrix-vector multiplication (SGEMV) for cuBLAS-like performance - A worklog – Maharshi's blogmaharshi.bearblog.dev
• The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
• Mini Project: GPU Accelerated Matrix Multiplication (almost) like cuBLAS0mean1sigma.com