Making Deep Learning Go Faster

So, you want to improve the performance of your deep learning model. How might you approach such a task? Often, folk fall back to a grab-bag of tricks that might've worked before or saw on a tweet. "Use in-place operations! Set gradients to None! Install PyTorch 1.10.0 but not 1.10.1!"

Making Deep Learning go Brrrr From First Principles Making Deep Learning Go Brrrr From First Principles So, you want to improve the performance of your deep learning model. How might you approach such a task? Often, folk fall back to a grab-bag of tricks that might've worked before or saw on a tweet. "Use in-place operations! Set gradients to None! Install PyTorch 1.10.0 but not 1.10.1!" It's understandable why users often take such an ad-hoc approach performance on modern systems (particularly deep learning) often feels as much like alchemy as it does science. That being said, reasoning from

saved by

• Feranno Liu
• Alex K. Chen
• Pranav
• Karan MJ
• Varun Shenoy
• Ratan Kaliani
• Fobi Boateng
• Shubham Chandel
• Benedict Neo
• Raffi Hotter
• Raayan Dhar
• Jianmin Chen

related reading

• Transformer Inference Arithmetic | kipply's blogkipp.ly
• How To Scale Your Modeljax-ml.github.io
• All About Rooflines | How To Scale Your Modeljax-ml.github.io
• The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• The Little Book of Deep Learningfleuret.org
• Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
• GPU Performance Background User's Guide - NVIDIA Docsdocs.nvidia.com
• Estimating training compute of deep learning models | Epoch AIepochai.org
• The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
• How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
• How to Think About TPUs | How To Scale Your Modeljax-ml.github.io