Human-like Neural Nets by Catapulting · Gwern.net
Speculative proposal to create artificial neural nets with human-like performance by high-learning-rate/regularization training of overparameterized NNs to trigger catapulting/grokking. Over-parameterization as a route to true generalization would resolve many outstanding mysteries of artificial versus natural intelligence.
--- title: Human-like Neural Nets by Catapulting description: "Speculative proposal to create artificial neural nets with human-like performance by high-learning-rate/regularization training of overparameterized NNs to trigger catapulting/grokking. Over-parameterization as a route to true generalization would resolve many outstanding mysteries of artificial versus natural intelligence." thumbnail: /doc/reinforcement-learning/model-free/2008-12-07-rogerjohansson-geneticprogrammingevolvingmonalisa-figure1-2x2subset-scalingoptimization3oomsshowsimprovementinrenderingmonalisawith50polygons.jpg thu
saved by
related reading
- Ilya Sutskever — We're moving from the age of scaling to the age of researchdwarkesh.com
- Will scaling work? - by Dwarkesh Patel - Dwarkesh Podcastdwarkesh.com
- The Scaling Hypothesis · Gwern.netgwern.net
- Andrej Karpathy — AGI is still a decade awaydwarkesh.com
- What Is ChatGPT Doing … and Why Does It Work?-Stephen Wolfram Writingswritings.stephenwolfram.com
- LLM Daydreaming · Gwern.netgwern.net
- On neural scaling and the quanta hypothesisericjmichaud.com
- Just Ask for Generalization | Eric Jangevjang.com
- Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetsmathai-iclr.github.io
- Will scaling work?dwarkeshpatel.com
- Do Machine Learning Models Memorize or Generalize?pair.withgoogle.com
- A Short History of Artificial Intelligenceevery.to