openai/parameter-golf: Train the smallest LM you can that fits in 16MB. Best model wins! ·
github.com · 3,968 words · saved by 1 readers
Train the smallest LM you can that fits in 16MB. Best model wins!
OpenAI Model Craft Challenge: Parameter Golf is a challenge to train the best language model that fits in a 16MB artifact and trains in under 10 minutes on 8xH100s, evaluated by compression on the FineWeb validation set (tokenizer-agnostic, bits per byte). This challenge is heavily inspired by the NanoGPT Speedrunning challenge, where participants compete to train a model that reaches 3.28 FineWeb validation loss as quickly as possible. We're excited to see how optimizing for a parameter-constrained setting pushes people toward unique architectures (test-time compute, aggressive parameter…
saved by
related reading
- Keller Jordan (@kellerjordan0) on Xx.com
- GitHub - karpathy/nanochat: The best ChatGPT that $100 can buy.github.com
- frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
- GitHub - karpathy/autoresearch: AI agents running research on single-GPU nanochat training automaticallygithub.com
- microgptkarpathy.github.io
- Training NanoGPT in 39.9 Secondshyperstition.cc
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- Let's build GPT: from scratch, in code, spelled out.youtube.com
- [2005.14165] Language Models are Few-Shot Learnersarxiv.org
- >10x More Efficient Pretraining — Magicmagic.dev
- [2509.14786] Pre-training under infinite computearxiv.org
- Evidence on AI R&D Progress from NanoGPT - METRmetr.org