Unpacking decentralized training - knower's substack
First things first, SPECIAL SHOUTOUT to sam lehman, rodeo, haus, yb, smac, ronan, and ibuyrugs for all of the comments, edits, feedback, and suggestions - you helped bring this to life and I really appreciate it. Also, some of these arxiv links open up as browser-based PDFs, so just a warning incase you don’t want to deal with that. As I’m writing this it’s been three months since my last post. What have I been up to since then? I don’t know. I’ve been doing a lot of reading, trying to workout like five days a week, and generally making the most of my last semester attending college. My mind gets a bit antsy anytime it’s been a month or two since I’ve written a long report, so this is my attempt at returning to baseline and getting back in the groove of things. If you couldn’t tell by the title, this is a report that’s largely about distributed/decentralized training, accompanied by some info covering what’s been happening in the world of AI, and some commentary about how all of this f
Unpacking decentralized training My attempt at highlighting the necessity of training LLMs in a distributed / decentralized manner, explaining a web of topics, and articulating the potential synergies between crypto + AI knower Apr 08, 2025 34 3 2 Share Where have I been? First things first, SPECIAL SHOUTOUT to sam lehman , rodeo , haus , yb , smac , ronan , and ibuyrugs for all of the comments, edits, feedback, and suggestions - you helped bring this to life and I really appreciate it. Also, some of these arxiv links open up as browser-based PDFs, so just a warning incase you don’t want to de
related reading
- How To Scale Your Modeljax-ml.github.io
- The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
- The Scaling Hypothesis · Gwern.netgwern.net
- The Short Case for Nvidia Stock | YouTube Transcript Optimizeryoutubetranscriptoptimizer.com
- As Rocks May Think | Eric Jangevjang.com
- Questions about the Future of AI - by Dwarkesh Pateldwarkesh.com
- Multi-Datacenter Training: OpenAI's Ambitious Plan To Beat Google's Infrastructuresemianalysis.com
- frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
- Keep the Tokens Flowing: Lessons from 16 Open-Source RL Librarieshuggingface.co
- The upcoming GPT-3 moment for RL | Mechanize, Inc.mechanize.work
- Training Imperativesdan.io
- The Little Book of Deep Learningfleuret.org