A Mechanistic Interpretability Analysis of Grokking - LessWrong

A significantly updated version of this work is now on Arxiv …

x A Mechanistic Interpretability Analysis of Grokking — LessWrong Best of LessWrong 2022 Interpretability (ML & AI) Lottery Ticket Hypothesis Machine Learning (ML) Grokking (ML) AI Curated 378 A Mechanistic Interpretability Analysis of Grokking by Neel Nanda , Tom Lieberum 15th Aug 2022 AI Alignment Forum Linkpost for colab.research.google.com 43 min read 48 378 Ω 106 A significantly updated version of this work is now on Arxiv and was published as a spotlight paper at ICLR 2023 aka, how the best way to do modular addition is with Discrete Fourier Transforms and trig identities If you don't wa

related reading

• A Mechanistic Interpretability Analysis of Grokking — AI Alignment Forumalignmentforum.org
• Do Machine Learning Models Memorize or Generalize?pair.withgoogle.com
• [2301.05217] Progress measures for grokking via mechanistic interpretabilityarxiv.org
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• In-context Learning and Induction Headstransformer-circuits.pub
• Towards Understanding Grokking: An Effective Theory of Representation Learningarxiv.org
• Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
• [2201.02177] Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetsarxiv.org
• Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetsmathai-iclr.github.io
• Ambiguous out-of-distribution generalization on an algorithmic task — LessWronglesswrong.com
• interpreting GPT: the logit lens — LessWronglesswrong.com
• [2602.16849] On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokkingarxiv.org