Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

N/A

1st Mathematical Reasoning in General Artificial Intelligence Workshop, ICLR 2021. G ROKKING : G ENERALIZATION B EYOND OVERFIT- TING ON S MALL A LGORITHMIC DATASETS Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin & Vedant Misra OpenAI {alethea,yburda,harri,ibab,vedant}@openai.com A BSTRACT In this paper we propose to study generalization of neural networks on small al- gorithmically generated datasets. In this setting, questions about data efficiency, memorization, generalization, and speed of…

saved by

• Enora ʕ ᄋ ᴥ ᄋʔ
• Noah T

related reading

• [2201.02177] Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetsarxiv.org
• Understanding deep learning requires rethinking generalizationarxiv.org
• Do Machine Learning Models Memorize or Generalize?pair.withgoogle.com
• Clare Lyle | What's grokking good for?clarelyle.com
• Just Ask for Generalization | Eric Jangevjang.com
• Scaling Laws, Carefully | Lil'Loglilianweng.github.io
• The Scaling Hypothesis · Gwern.netgwern.net
• arxiv.org/pdf/1805.08522arxiv.org
• A Recipe for Training Neural Networkskarpathy.github.io
• The Little Book of Deep Learningfleuret.org
• [2503.02113] Deep Learning is Not So Mysterious or Differentarxiv.org
• A Theory of Deep Learning | Elements of a Vector Spaceelonlit.com