Algorithmic Improvement Is Probably Faster Than Scaling Now — LessWrong

lesswrong.com · 3,314 words · saved by 1 readers

Back in 2020, a group at OpenAI ran a conceptually simple test to quantify how much AI progress was attributable to algorithmic improvements. They took ImageNet models which were state-of-the-art at various times between 2012 and 2020, and checked how much compute was needed to train each to the level of AlexNet (the state-of-the-art from 2012). Main finding: over ~7 years, the compute required fell by ~44x. In other words, algorithmic progress yielded a compute-equivalent doubling time of ~16 months (though error bars are large in both directions). On the compute side of things, in 2018 a group at OpenAI estimated that the compute spent on the largest training runs was growing exponentially with a doubling rate of ~3.4 months, between 2012 and 2018. So at the time, the rate of improvement from compute scaling was much faster than the rate of improvement from algorithmic progress. (Though algorithmic improvement was still faster than Moore's Law; the compute increases were mostly drive

x Algorithmic Improvement Is Probably Faster Than Scaling Now — LessWrong AI Frontpage 147 Algorithmic Improvement Is Probably Faster Than Scaling Now by johnswentworth 6th Jun 2023 AI Alignment Forum 2 min read 25 147 Ω 49 The Story as of ~4 Years Ago Back in 2020, a group at OpenAI ran a conceptually simple test to quantify how much AI progress was attributable to algorithmic improvements. They took ImageNet models which were state-of-the-art at various times between 2012 and 2020, and checked how much compute was needed to train each to the level of AlexNet (the state-of-the-art from 2012).

Algorithmic Improvement Is Probably Faster Than Scaling Now — LessWrong

related reading