A Mechanistic Interpretability Analysis of Grokking - LessWrong

A significantly updated version of this work is now on Arxiv …

x A Mechanistic Interpretability Analysis of Grokking — LessWrong Best of LessWrong 2022 Interpretability (ML & AI) Lottery Ticket Hypothesis Machine Learning (ML) Grokking (ML) AI Curated 378 A Mechanistic Interpretability Analysis of Grokking by Neel Nanda , Tom Lieberum 15th Aug 2022 AI Alignment Forum Linkpost for colab.research.google.com 43 min read 48 378 Ω 106 A significantly updated version of this work is now on Arxiv and was published as a spotlight paper at ICLR 2023 aka, how the best way to do modular addition is with Discrete Fourier Transforms and trig identities If you don't wa

related reading

• A Mechanistic Interpretability Analysis of Grokking — AI Alignment Forumalignmentforum.org
• Do Machine Learning Models Memorize or Generalize?pair.withgoogle.com
• [2301.05217] Progress measures for grokking via mechanistic interpretabilityarxiv.org
• Clare Lyle | What's grokking good for?clarelyle.com
• [2301.05217] Progress measures for grokking via mechanistic interpretabilityarxiv.org
• Transformer Circuits Threadtransformer-circuits.pub
• Zipfian grokking | Jasper Gilleyjagilley.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• In-context Learning and Induction Headstransformer-circuits.pub
• Grokking (machine learning) - Wikipediaen.wikipedia.org
• Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
• On neural scaling and the quanta hypothesisericjmichaud.com