A Hitchhiker’s Guide to ML Training Infrastructure | CMU Software Engineering Institute
Hardware is a key enabler for machine learning. Recent advances in the field, including the introduction of graphics processing units, have had a significant impact on the training of AI systems.
A Hitchhiker’s Guide to ML Training Infrastructure Jay Palat February 21, 2022 PUBLISHED IN Artificial Intelligence Engineering CITE Get Citation SHARE This post has been shared 11 times. Hardware has made a huge impact on the field of machine learning (ML). Many of the ideas we use today were published decades ago, but the cost to run them and the data necessary were too expensive, making them impractical. Recent advances, including the introduction of graphics processing units (GPUs), are making some of those ideas a reality. In this post we’ll look at some of the hardware factors that impac
saved by
related reading
- How To Scale Your Modeljax-ml.github.io
- Making Deep Learning go Brrrr From First Principleshorace.io
- The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
- A friendly introduction to machine learning compilers and optimizershuyenchip.com
- How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
- The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
- Training great LLMs entirely from ground up in the wilderness as a startup - Yi Tayyitay.net
- The Hardware Lotteryhardwarelottery.github.io
- Multi-Datacenter Training: OpenAI's Ambitious Plan To Beat Google's Infrastructuresemianalysis.com
- Horace He: Building Machine Learning Systems for a Trillion Trillion Floating Point Operationsyoutube.com
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- Estimating training compute of deep learning models | Epoch AIepochai.org