Why GEMM is at the heart of deep learning « Pete Warden's blog
Photo by Anthony Catalano I spend most of my time worrying about how to make deep learning with neural networks faster and more power efficient. In practice that means focusing on a function called…
Photo by Anthony Catalano I spend most of my time worrying about how to make deep learning with neural networks faster and more power efficient. In practice that means focusing on a function called GEMM. It’s part of the BLAS (Basic Linear Algebra Subprograms) library that was first created in 1979, and until I started trying to optimize neural networks I’d never heard of it. To explain why it’s so important, here’s a diagram from my friend Yangqing Jia’s thesis: This is breaking down where the time’s going for a typical deep convolutional neural network doing image recognition using Alex…
saved by
related reading
- How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklogsiboehm.com
- Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com
- CS231n Deep Learning for Computer Visioncs231n.github.io
- A Gentle Introduction to Graph Neural Networksdistill.pub
- Making Deep Learning go Brrrr From First Principleshorace.io
- Worklog: Optimising GEMM on NVIDIA H100 for cuBLAS-like Performance (WIP) – Hamza's Bloghamzaelshafie.bearblog.dev
- Convolutional Neural Networks, Explained | Towards Data Sciencetowardsdatascience.com
- A Beginner's Guide To Understanding Convolutional Neural Networks – Adit Deshpande – Engineering at Forward | UCLA CS '19adeshpande3.github.io
- Understanding Convolutions on Graphsdistill.pub
- Neural Networks, Types, and Functional Programming -- colah's blogcolah.github.io
- ImageNet Classification with Deep Convolutional Neural Networksproceedings.neurips.cc
- The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com