A Call to Build Models Like We Build Open-Source Software

1  Introduction 2  A brief explanation of open-source software development 3  Community-developed and continually-improved models   3.1  Incremental and cheaply-communicable updates   3.2  Merging models   3.3  Vetting community contributions   3.4  Versioning and backward compatibility   3.5  Modularity and distribution 4  An example future 5  Conclusion

A Call to Build Models Like We Build Open-Source Software A Call to Build Models Like We Build Open-Source Software Colin Raffel December 8th, 2021 colinraffel.com/blog Contents 1 Introduction 2 A brief explanation of open-source software development 3 Community-developed and continually-improved models 3.1 Incremental and cheaply-communicable updates 3.2 Merging models 3.3 Vetting community contributions 3.4 Versioning and backward compatibility 3.5 Modularity and distribution 4 An example future 5 Conclusion Introduction Over the past decade or so, it has become increasingly common to use tr

saved by

• Varun Shenoy
• Noah T
• Ishan Mukherjee

related reading

• Google "We Have No Moat, And Neither Does OpenAI"semianalysis.com
• Recent Advances in Language Model Fine-tuningruder.io
• What I learned this week - Can distillation be stopped, Mythos and the cybersecurity equilibrium, Pipeline RLdwarkesh.com
• Composer2.pdfcursor.com
• Introducing Marin: An Open Lab for Building Foundation Models | Marinmarin.community
• Why We Need Continual Learning | Andreessen Horowitza16z.com
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• What comes next with open models - by Nathan Lambertinterconnects.ai
• MAI-Thinking-1: Building a Hill-Climbing Machinemicrosoft.ai
• What's so hard about continuous learning?seangoedecke.com
• Model Merging — a biased overview | GLADIAgladia-research-group.github.io
• Machine learning needs better tools – Replicate blogreplicate.com