Open Weights isn't Open Training

How many monkey-patches does it take to post-train a trillion parameter model?

When I was in college, my data structures professor told a story. It went something like this: "When I was your age, I received an assignment, and encountered an inexplicable bug. I debugged and debugged and found that adding a print statement resolved the bug. I was young like all of you, and I was certain I'd found a bug in the C compiler. Turns out the problem was me." The takeaway was clear: if you have a bug, it's your fault. This is a good heuristic for most cases, but with open source ML infrastructure, you need to throw this advice out the window. There might be features that…

saved by

• Karthik Suresh

related reading

• Making Deep Learning go Brrrr From First Principleshorace.io
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• A Call to Build Models Like We Build Open-Source Softwarecolinraffel.com
• A Recipe for Training Neural Networkskarpathy.github.io
• A Recipe for Training Neural Networkskarpathy.github.io
• PiTorch: ML on Baremetal Raspberry Pis | projectsmasonjwang.com
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• the bug that taught me more about PyTorch than years of using it | Elana Simonelanapearl.github.io
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• Keep the Tokens Flowing: Lessons from 16 Open-Source RL Librarieshuggingface.co
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• Training great LLMs entirely from ground up in the wilderness as a startup - Yi Tayyitay.net