frontier model training methodologies | Alex Wa’s Blog
How do labs train a frontier, multi-billion parameter model? We look towards seven open-weight frontier models: Hugging Face’s SmolLM3, Prime Intellect’s Intellect 3, Nous Research’s Hermes 4, OpenAI’s gpt-oss-120b, Moonshot’s Kimi K2, DeepSeek’s DeepSeek-R1, and Arcee’s Trinity series. This blog is an attempt at distilling the techniques, motivations, and considerations used to train their models with an emphasis on training methodology over infrastructure.
Share on: How do labs train a frontier, multi-billion parameter model? We look towards seven open-weight frontier models: Hugging Face’s SmolLM3 , Prime Intellect’s Intellect 3 , Nous Research’s Hermes 4 , OpenAI’s gpt-oss-120b , Moonshot’s Kimi K2 , DeepSeek’s DeepSeek-R1 , and Arcee’s Trinity series . This blog is an attempt at distilling the techniques, motivations, and considerations used to train their models with an emphasis on training methodology over infrastructure. These notes are largely structured based on Hugging Face’s SmolLM3 report due to its extensiveness, and it is currently
saved by
- Karan MJ
- Freeman Jiang
- Rishi Kothari
- Benedict Neo
- Akira Yoshiyama
- Denys
- Vihan Tiwari
- Ishaan Panigrahi
- Sam Wang
- Miguel Villa Floran
related reading
- The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
- Is Frontier Asynchronous RL Solved? — Luke J. Huangluk-huang.github.io
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- How Kimi Trains 896 Experts Without Collapseyoutube.com
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
- How trillion-parameter AI models work and how to serve themyoutube.com
- ali (@waterloo_intern) on Xx.com
- >10x More Efficient Pretraining — Magicmagic.dev
- Training great LLMs entirely from ground up in the wilderness as a startup - Yi Tayyitay.net
- Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
- A History of Large Language Modelsgregorygundersen.com