AI scaling myths
normaltech.ai · 2,326 words · saved by 1 readers
Scaling will run out. The question is when.
So far, bigger and bigger language models have proven more and more capable. But does the past predict the future? One popular view is that we should expect the trends that have held so far to continue for many more orders of magnitude, and that it will potentially get us to artificial general intelligence, or AGI. This view rests on a series of myths and misconceptions. The seeming predictability of scaling is a misunderstanding of what research has shown. Besides, there are signs that LLM developers are already at the limit of high-quality training data. And the industry is seeing strong…
related reading
- Will scaling work?dwarkeshpatel.com
- Will scaling work? - by Dwarkesh Patel - Dwarkesh Podcastdwarkesh.com
- Scaling: The State of Play in AIoneusefulthing.org
- Scaling Laws, Carefully | Lil'Loglilianweng.github.io
- The Scaling Hypothesis · Gwern.netgwern.net
- Scaling is subtler than it seemsberen.io
- Ilya Sutskever — We're moving from the age of scaling to the age of researchdwarkesh.com
- Is AI progress slowing down?normaltech.ai
- AI progress is about to speed up | Epoch AIepoch.ai
- Dario Amodei — "We are near the end of the exponential"dwarkesh.com
- My picture of the present in AI — LessWronglesswrong.com
- The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com