[Hero Run] 535B-A23B on 18T tokens · Issue #8435 · marin-community/marin
Issue is to describe the Hero Run scaling ladder, config, and track its performance. Tracking: https://wandb.ai/marin-community/marin_moe/reports/535B-A23B-18T-Token-Hero-Run-Scaling-Ladder--Vmlldz...
Issue is to describe the Hero Run scaling ladder, config, and track its performance. Tracking: https://wandb.ai/marin-community/marin_moe/reports/535B-A23B-18T-Token-Hero-Run-Scaling-Ladder--VmlldzoxNzc2MDM5Ng Details on the full model spec is in the comment below. Why run a scaling ladder? This lets us project the model performance and compare it to the scaling projections of our prior recipes. If our projection looks meaningfully worse, then we know we need to go back to the drawing board on some aspect of the model or data. This step can catch a lot of bugs! We can look at how…
saved by
related reading
- Mixture of Experts Quantile Balancing: Validated at 32B-A5B (1e22 FLOPs) Scaleopenathena.ai
- Scaling Laws That Extrapolate 300× Past the Fitopenathena.ai
- [2509.14786] Pre-training under infinite computearxiv.org
- A short note on some aspects of long context attention | nor's blognor-blog.pages.dev
- Scaling Laws, Carefully | Lil'Loglilianweng.github.io
- Marin 535B-A23B launch noteopenathena.ai
- The Scaling Hypothesis · Gwern.netgwern.net
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
- IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
- Scaling is subtler than it seemsberen.io
- Keep the Tokens Flowing: Lessons from 16 Open-Source RL Librarieshuggingface.co