Making Deep Learning Go Faster

So, you want to improve the performance of your deep learning model. How might you approach such a task? Often, folk fall back to a grab-bag of tricks that might've worked before or saw on a tweet. "Use in-place operations! Set gradients to None! Install PyTorch 1.10.0 but not 1.10.1!"

Making Deep Learning go Brrrr From First Principles Making Deep Learning Go Brrrr From First Principles So, you want to improve the performance of your deep learning model. How might you approach such a task? Often, folk fall back to a grab-bag of tricks that might've worked before or saw on a tweet. "Use in-place operations! Set gradients to None! Install PyTorch 1.10.0 but not 1.10.1!" It's understandable why users often take such an ad-hoc approach performance on modern systems (particularly deep learning) often feels as much like alchemy as it does science. That being said, reasoning from

saved by

• Feranno Liu
• Alex K. Chen
• Pranav
• Karan MJ
• Varun Shenoy
• Ratan Kaliani
• Fobi Boateng
• Shubham Chandel
• Benedict Neo
• Raffi Hotter
• Raayan Dhar
• Jianmin Chen

related reading

• How To Scale Your Modeljax-ml.github.io
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• All About Rooflines | How To Scale Your Modeljax-ml.github.io
• Horace He: Building Machine Learning Systems for a Trillion Trillion Floating Point Operationsyoutube.com
• A friendly introduction to machine learning compilers and optimizershuyenchip.com
• A Recipe for Training Neural Networkskarpathy.github.io
• The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
• A Recipe for Training Neural Networkskarpathy.github.io
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• PiTorch: ML on Baremetal Raspberry Pis | projectsmasonjwang.com
• The Little Book of Deep Learningfleuret.org