Strangely, Matrix Multiplications on GPUs Run Faster When Given "Predictable" Data! [short]

Great minds discuss flops per watt.

Strangely, Matrix Multiplications on GPUs Run Faster When Given "Predictable" Data! [short] Great minds discuss flops per watt. Horace He Apr 29, 2024 170 21 10 Share It’s 2022. I check out this cool new project, CUTLASS , with very fast matmuls. I take a large matmul, 8192 x 8192 x 8192, and benchmark it in PyTorch, which calls CuBLAS. python mm_bench.py > CuBLAS: 258 Teraflops Not bad, 83% flop utilization. Now let’s check out Cutlass’s performance using their profiler. ./cutlass_profiler --operation=Gemm --m=8192 --n=8192 --k=8192 > CUTLASS: 288 Teraflops !!! 10% higher perf? That’s incredi

saved by

• Neel Redkar
• Sudarsh K
• fernando s
• Ashray Gupta

related reading

• How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklogsiboehm.com
• Making Deep Learning go Brrrr From First Principleshorace.io
• Outperforming cuBLAS on H100: a Worklogcudaforfun.substack.com
• Outperforming cuBLAS on H100: a Worklogcudaforfun.substack.com
• Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com
• The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
• What happens when you run a CUDA kernelfergusfinn.com
• All About Rooflines | How To Scale Your Modeljax-ml.github.io
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Horace He: Building Machine Learning Systems for a Trillion Trillion Floating Point Operationsyoutube.com
• What Shapes Do Matrix Multiplications Like? [medium]thonking.ai
• The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com