Being GPU Poor makes you creative

I started my sabbatical pretty GPU poor. Going from “whatever compute I needed at Apple” to “my MacBook” was a real drop. Then I walked into the Recurse Center hub in NYC for the first time, and so

I started my sabbatical pretty GPU poor. Going from “whatever compute I needed at Apple” to “my MacBook” was a real drop. Then I walked into the Recurse Center hub in NYC for the first time, and someone casually mentioned they had “a couple of GPUs” sitting around. That got my attention. The catch was that the hardware was not homogeneous. RC had a couple of NVIDIA GeForce GTX TITAN X cards, and I’d brought my M4 Max MacBook with Apple Silicon. Different vendors, different drivers, different memory architectures, different runtimes. The classic “how do I get all of these to cooperate?” problem

saved by

• C LI

related reading

• The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
• How To Scale Your Modeljax-ml.github.io
• Multi-Datacenter Training: OpenAI's Ambitious Plan To Beat Google's Infrastructuresemianalysis.com
• How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
• Scale Machine Learning & AI Computing | Ray by Anyscaleray.io
• Training great LLMs entirely from ground up in the wilderness as a startup - Yi Tayyitay.net
• Components of an Open Source AI Compute Tech Stackanyscale.com
• The Short Case for Nvidia Stock | YouTube Transcript Optimizeryoutubetranscriptoptimizer.com
• The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
• PiTorch: ML on Baremetal Raspberry Pis | projectsmasonjwang.com
• Parallelism in Distributed Deep Learning · Better Tomorrow with Computer Scienceinsujang.github.io
• Pipeline-Parallelism: Distributed Training via Model Partitioningsiboehm.com