Pre-training under infinite compute

N/A

Pre-training under infinite compute Konwoo Kim∞ , Suhas Kotha∞ , Percy Liang, Tatsunori Hashimoto Stanford University Abstract arXiv:2509.14786v1 [cs.LG] 18 Sep 2025 Since compute grows much faster than web text available for language model pre-training, we ask how one…

related reading

• [2509.14786] Pre-training under infinite computearxiv.org
• [2509.14786] Pre-training under infinite computearxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• Scaling Laws, Carefully | Lil'Loglilianweng.github.io
• [2605.12715] Scaling Laws for Mixture Pretraining Under Data Constraintsarxiv.org
• IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
• [2605.06546] Efficient Pre-Training with Token Superpositionarxiv.org
• >10x More Efficient Pretraining — Magicmagic.dev
• Compression and Intelligencegreene.sh
• [2512.23675] End-to-End Test-Time Training for Long Contextarxiv.org
• Pretraining progress is mostly coming from datadwarkesh.com
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org