Will scaling work? - by Dwarkesh Patel - Dwarkesh Podcast
If we can keep scaling LLMs++ (and get better and more general performance as a result), then there’s reason to expect powerful AIs by 2040 (or much sooner) which can automate most cognitive labor and speed up further AI progress. However, if scaling doesn’t work, then the path to AGI seems much longer and more intractable, for reasons I explain in the post. In order to think through both the pro and con arguments about scaling, I wrote the post as a debate between two characters I made up - Believer and Skeptic. Skeptic: We’re about to run out of high quality language data next year. Even taking handwavy scaling curves seriously implies that we’ll need 1e35 FLOPs for an AI that is reliable and smart enough to write a scientific paper (that’s table stakes for the abilities an AI would need to automate further AI research and continue progress once scaling becomes infeasible)1. Which means we need 5 OOMs (orders of magnitude) more data than we seem to have2. I’m worried that when people
Explore this link on the map →