Human-like Neural Nets by Catapulting · Gwern.net
Speculative proposal to create artificial neural nets with human-like performance by high-learning-rate/regularization training of overparameterized NNs to trigger catapulting/grokking. Over-parameterization as a route to true generalization would resolve many outstanding mysteries of artificial versus natural intelligence.
--- title: Human-like Neural Nets by Catapulting description: "Speculative proposal to create artificial neural nets with human-like performance by high-learning-rate/regularization training of overparameterized NNs to trigger catapulting/grokking. Over-parameterization as a route to true generalization would resolve many outstanding mysteries of artificial versus natural intelligence." thumbnail: /doc/reinforcement-learning/model-free/2008-12-07-rogerjohansson-geneticprogrammingevolvingmonalisa-figure1-2x2subset-scalingoptimization3oomsshowsimprovementinrenderingmonalisawith50polygons.jpg thu
saved by
related reading
- The Scaling Hypothesis · Gwern.netgwern.net
- Guardian Angels: LLM Personalization for Productivity and Security · Gwern.netgwern.net
- Why Tool AIs Want to Be Agent AIs · Gwern.netgwern.net
- The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
- You can’t imitation-learn how to continual-learn — LessWronglesswrong.com
- Writing for LLMs So They Listen · Gwern.netgwern.net
- The World Inside Neural Networksgoodfire.ai
- On neural scaling and the quanta hypothesisericjmichaud.com
- LLM Daydreaming · Gwern.netgwern.net
- What Is ChatGPT Doing … and Why Does It Work?-Stephen Wolfram Writingswritings.stephenwolfram.com
- Using Artificial Intelligence to Augment Human Intelligencedistill.pub
- A Recipe for Training Neural Networkskarpathy.github.io