✳flâneur — a map of the web's best reading

[2507.10618] Compute Requirements for Algorithmic Innovation in Frontier AI Models

arxiv.org · 624 words · saved by 1 readers

Abstract:Algorithmic innovation in the pretraining of large language models has driven a massive reduction in the total compute required to reach a given level of capability. In this paper we empirically investigate the compute requirements for developing algorithmic innovations. We catalog 36 pre-training algorithmic innovations used in Llama 3 and DeepSeek-V3. For each innovation we estimate both the total FLOP used in development and the FLOP/s of the hardware utilized. Innovations using significant resources double in their requirements each year. We then use this dataset to investigate the effect of compute caps on innovation. Our analysis suggests that compute caps alone are unlikely to dramatically slow AI algorithmic progress. Even stringent compute caps -- such as capping total operations to the compute used to train GPT-2 or capping hardware capacity to 8 H100 GPUs -- could still have allowed for half of the cataloged innovations.

[2507.10618] Compute Requirements for Algorithmic Innovation in Frontier AI Models Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Machine Learning arXiv:2507.10618 (cs) [Submitted on 13 Jul 2025] Title: Compute Requirements for Algorithmic Innovation in Frontier AI Models Authors: Peter Barnett View a PDF of the paper titled Compute Requirements for Algorithmic Innovation in Frontier AI Models, by Peter Barnett View PDF HTML (experimental) Abstract: Algorithmic innovation in the p

Explore this link on the map →

related reading