Human-like Neural Nets by Catapulting · Gwern.net
Speculative proposal to create artificial neural nets with human-like performance by high-learning-rate/regularization training of overparameterized NNs to trigger catapulting/grokking. Over-parameterization as a route to true generalization would resolve many outstanding mysteries of artificial versus natural intelligence.
--- title: Human-like Neural Nets by Catapulting description: "Speculative proposal to create artificial neural nets with human-like performance by high-learning-rate/regularization training of overparameterized NNs to trigger catapulting/grokking. Over-parameterization as a route to true generalization would resolve many outstanding mysteries of artificial versus natural intelligence." thumbnail: /doc/reinforcement-learning/model-free/2008-12-07-rogerjohansson-geneticprogrammingevolvingmonalisa-figure1-2x2subset-scalingoptimization3oomsshowsimprovementinrenderingmonalisawith50polygons.jpg thu
Explore this link on the map →saved by
related reading
- Will scaling work? - by Dwarkesh Patel - Dwarkesh Podcastdwarkesh.com
- The Scaling Hypothesis · Gwern.netgwern.net
- Andrej Karpathy — AGI is still a decade awaydwarkesh.com
- What Is ChatGPT Doing … and Why Does It Work?-Stephen Wolfram Writingswritings.stephenwolfram.com
- LLM Daydreaming · Gwern.netgwern.net
- On neural scaling and the quanta hypothesisericjmichaud.com
- Just Ask for Generalization | Eric Jangevjang.com
- Clare Lyle | What's grokking good for?clarelyle.com
- What If We Had Bigger Brains? Imagining Minds beyond Ours-Stephen Wolfram Writingswritings.stephenwolfram.com
- AI Timelines via Cumulative Optimization Power: Less Long, More Short — LessWronglesswrong.com
- Building machines that learn and think like people | Behavioral and Brain Sciences | Cambridge Corecambridge.org
- A Short History of Artificial Intelligenceevery.to