All About Rooflines | How To Scale Your Model
When we run algorithms on hardware, we're bounded by three things: how fast our computer can do math (OPs/second), the bandwidth available for moving data around (bytes/second), and the total memory available to store data (bytes). These
All About Rooflines | How To Scale Your Model All About Rooflines Part 1 of How To Scale Your Model ( Part 0: Introduction | Part 2: TPUs ) When we run algorithms on hardware, we're bounded by three things: how fast our computer can do math (OPs/second), the bandwidth available for moving data around (bytes/second), and the total memory available to store data (bytes). These "roofline" constraints let us upper and lower bound the time of a given computation. Authors Affiliation Jacob Austin Google DeepMind Sholto Douglas Roy Frostig Anselm Levskaya Charlie Chen Sharad Vikram Federico Lebron Pe
Explore this link on the map →saved by
- Jennifer Zhao
- Dron Hazra
- Clementine Yang
- Khalid Ali
- Nathan Chen
- LIME YAO
- Rohan Kanti
- Ishan Mukherjee
- Ali Ahmad
related reading
- How to Think About TPUs | How To Scale Your Modeljax-ml.github.io
- How To Scale Your Modeljax-ml.github.io
- Making Deep Learning go Brrrr From First Principleshorace.io
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
- GPU Performance Background User's Guide - NVIDIA Docsdocs.nvidia.com
- CVPR2023_eff_tutorial_molchanov.pdfnvlabs.github.io
- Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com
- All the Transformer Math You Need to Know | How To Scale Your Modeljax-ml.github.io
- 1.5x faster MoE training with custom MXFP8 kernels · Cursorcursor.com
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- Worklog: Optimising GEMM on NVIDIA H100 for cuBLAS-like Performance (WIP) – Hamza's Bloghamzaelshafie.bearblog.dev