GPU Glossary

We wrote this glossary to solve a problem we ran into working with GPUs here at Modal : the documentation is fragmented, making it difficult to connect concepts at different levels of the stack, like Streaming Multiprocessor Architecture , Compute Capability , and nvcc compiler flags . So we've read the PDFs from NVIDIA , lurked in the good Discords , and even bought dead-tree textbooks to put together a glossary that spans the whole stack in one place. This glossary, unlike a PDF or a Discord or a book, is a hypertext document -- all pages are inter-linked with one another, so you can jump down to read about the Warp Scheduler so you can better understand the threads that you came across in the article on the CUDA programming model . You can also read it linearly. To navigate between pages, use the arrow keys, the arrows at the bottom of each page, or the table of contents (in the sidebar on desktop or in the hamburger menu on mobile). The source for the glossary is available on Gi

██████╗ ██████╗ ██╗ ██╗ ██╔════╝ ██╔══██╗██║ ██║ ██║ ███╗██████╔╝██║ ██║ ██║ ██║██╔═══╝ ██║ ██║ ╚██████╔╝██║ ╚██████╔╝ ╚═════╝ ╚═╝ ╚═════╝ ██████╗ ██╗ ██████╗ ███████╗███████╗ █████╗ ██████╗ ██╗ ██╗ ██╔════╝ ██║ ██╔═══██╗██╔════╝██╔════╝██╔══██╗██╔══██╗╚██╗ ██╔╝ ██║ ███╗██║ ██║ ██║███████╗███████╗███████║██████╔╝ ╚████╔╝ ██║ ██║██║ ██║ ██║╚════██║╚════██║██╔══██║██╔══██╗ ╚██╔╝ ╚██████╔╝███████╗╚██████╔╝███████║███████║██║ ██║██║ ██║ ██║ ╚═════╝ ╚══════╝ ╚═════╝ ╚══════╝╚══════╝╚═╝ ╚═╝╚═╝ ╚═╝ ╚═╝ We wrote this glossary to solve a problem we ran into working with GPUs here at Modal : the…

saved by

• Rajan Agarwal
• Annie Zhou
• Chloe Yan
• Liam Hinzman
• Vincent Huang
• Ishan Mukherjee
• Denys
• Caden Juang
• SUJASH AGRAWAL
• Yash Karthik

related reading

• CUDA Programming Guide — CUDA Programming Guidedocs.nvidia.com
• How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
• Demystifying GPU Compute Architectures - by Babbagethechipletter.substack.com
• GitHub - gpu-mode/resource-stream: GPU programming related news and material linksgithub.com
• CUDA C++ Programming Guide (Legacy) — CUDA C++ Programming Guidedocs.nvidia.com
• A case for learning GPU programming with a compute-first mindset – Maister's Graphics Adventuresthemaister.net
• Demystifying GPU Compute Architecturesthechipletter.substack.com
• Implementing a GPU's Programming Model on a CPUlitherum.blogspot.com
• Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com
• GitHub - adam-maj/tiny-gpu: A minimal GPU design in Verilog to learn how GPUs work from the ground up · GitHubgithub.com
• What happens when you run a CUDA kernelfergusfinn.com
• GitHub - wafer-ai/gpu-perf-engineering-resources: A curated resource list for learning AI performance engineering, from GPU fundamentals to production inference.github.com