AI scaling myths
normaltech.ai · 2,326 words · saved by 1 readers
Scaling will run out. The question is when.
So far, bigger and bigger language models have proven more and more capable. But does the past predict the future? One popular view is that we should expect the trends that have held so far to continue for many more orders of magnitude, and that it will potentially get us to artificial general intelligence, or AGI. This view rests on a series of myths and misconceptions. The seeming predictability of scaling is a misunderstanding of what research has shown. Besides, there are signs that LLM developers are already at the limit of high-quality training data. And the industry is seeing strong…
related reading
- Will scaling work?dwarkeshpatel.com
- Scaling: The State of Play in AIoneusefulthing.org
- Will scaling work? - by Dwarkesh Patel - Dwarkesh Podcastdwarkesh.com
- AI 2027ai-2027.com
- Is AI progress slowing down?normaltech.ai
- I. From GPT-4 to AGI: Counting the OOMs - SITUATIONAL AWARENESSsituational-awareness.ai
- The Scaling Hypothesis · Gwern.netgwern.net
- There Are No New Ideas in AI… Only New Datasetsblog.jxmo.io
- Scaling is subtler than it seemsberen.io
- The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
- Why do people disagree about when powerful AI will arrive?blog.bluedot.org
- Can AI scaling continue through 2030? | Epoch AIepoch.ai