Making Deep Learning Go Faster
So, you want to improve the performance of your deep learning model. How might you approach such a task? Often, folk fall back to a grab-bag of tricks that might've worked before or saw on a tweet. "Use in-place operations! Set gradients to None! Install PyTorch 1.10.0 but not 1.10.1!"
Making Deep Learning go Brrrr From First Principles Making Deep Learning Go Brrrr From First Principles So, you want to improve the performance of your deep learning model. How might you approach such a task? Often, folk fall back to a grab-bag of tricks that might've worked before or saw on a tweet. "Use in-place operations! Set gradients to None! Install PyTorch 1.10.0 but not 1.10.1!" It's understandable why users often take such an ad-hoc approach performance on modern systems (particularly deep learning) often feels as much like alchemy as it does science. That being said, reasoning from
Explore this link on the map →saved by
- Alex K. Chen
- Pranav
- Karan MJ
- Varun Shenoy
- Ratan Kaliani
- Fobi Boateng
- Shubham Chandel
- Benedict Neo
- Raffi Hotter
- Raayan Dhar
- Kevin Wang
- Sebastian J Zhao
related reading
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- How To Scale Your Modeljax-ml.github.io
- All About Rooflines | How To Scale Your Modeljax-ml.github.io
- The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- The Little Book of Deep Learningfleuret.org
- GPU Performance Background User's Guide - NVIDIA Docsdocs.nvidia.com
- Estimating training compute of deep learning models | Epoch AIepochai.org
- A friendly introduction to machine learning compilers and optimizershuyenchip.com
- How to Think About TPUs | How To Scale Your Modeljax-ml.github.io
- How to Think About GPUs | How To Scale Your Modeljax-ml.github.io