Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

N/A

1st Mathematical Reasoning in General Artificial Intelligence Workshop, ICLR 2021. G ROKKING : G ENERALIZATION B EYOND OVERFIT- TING ON S MALL A LGORITHMIC DATASETS Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin & Vedant Misra OpenAI {alethea,yburda,harri,ibab,vedant}@openai.com A BSTRACT In this paper we propose to study generalization of neural networks on small al- gorithmically generated datasets. In this setting, questions about data efficiency, memorization, generalization, and speed of…

saved by

• Enora ʕ ᄋ ᴥ ᄋʔ
• Noah T

related reading

• [2201.02177] Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetsarxiv.org
• Do Machine Learning Models Memorize or Generalize?pair.withgoogle.com
• Just Ask for Generalization | Eric Jangevjang.com
• Understanding deep learning requires rethinking generalizationarxiv.org
• Towards Understanding Grokking: An Effective Theory of Representation Learningarxiv.org
• The Scaling Hypothesis · Gwern.netgwern.net
• [2605.01172] A Theory of Generalization in Deep Learningarxiv.org
• Ambiguous out-of-distribution generalization on an algorithmic task — LessWronglesswrong.com
• arxiv.org/pdf/1805.08522arxiv.org
• Clare Lyle | What's grokking good for?clarelyle.com
• Hypothesis: gradient descent prefers general circuits — LessWronglesswrong.com
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org