Being GPU Poor makes you creative

I started my sabbatical pretty GPU poor. Going from “whatever compute I needed at Apple” to “my MacBook” was a real drop. Then I walked into the Recurse Center hub in NYC for the first time, and so

I started my sabbatical pretty GPU poor. Going from “whatever compute I needed at Apple” to “my MacBook” was a real drop. Then I walked into the Recurse Center hub in NYC for the first time, and someone casually mentioned they had “a couple of GPUs” sitting around. That got my attention. The catch was that the hardware was not homogeneous. RC had a couple of NVIDIA GeForce GTX TITAN X cards, and I’d brought my M4 Max MacBook with Apple Silicon. Different vendors, different drivers, different memory architectures, different runtimes. The classic “how do I get all of these to cooperate?” problem

saved by

• C LI

related reading

• How To Scale Your Modeljax-ml.github.io
• The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
• How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
• Multi-Datacenter Training: OpenAI's Ambitious Plan To Beat Google's Infrastructuresemianalysis.com
• From Single-Node to Multi-GPU Clusters: How Discord Made Distributed Compute Easy for ML Engineersdiscord.com
• Scale Machine Learning & AI Computing | Ray by Anyscaleray.io
• Keep the Tokens Flowing: Lessons from 16 Open-Source RL Librarieshuggingface.co
• The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
• Training great LLMs entirely from ground up in the wilderness as a startup - Yi Tayyitay.net
• Components of an Open Source AI Compute Tech Stackanyscale.com
• Pipeline-Parallelism: Distributed Training via Model Partitioningsiboehm.com
• The Short Case for Nvidia Stock | YouTube Transcript Optimizeryoutubetranscriptoptimizer.com