Investigating the learning coefficient of modular addition: hackathon project — LessWrong

As our project at the Melbourne hackathon on Singular Learning Theory and alignment (Oct. 7-8), we did some experiments to estimate the learning coef…

x Investigating the learning coefficient of modular addition: hackathon project — LessWrong Singular Learning Theory Interpretability (ML & AI) AI Frontpage 97 Investigating the learning coefficient of modular addition: hackathon project by Nina Panickssery , Dmitry Vaintrob 17th Oct 2023 AI Alignment Forum 15 min read 5 97 Ω 42 As our project at the Melbourne hackathon on Singular Learning Theory and alignment (Oct. 7-8), we did some experiments to estimate the learning coefficient of the single-layer modular addition task at a basin, an invariant that measures the information complexity (rea

saved by

• Asher P
• Sam B

related reading

• DSLT 0. Distilling Singular Learning Theory — LessWronglesswrong.com
• The Local Learning Coefficient: A Singularity-Aware Complexity Measurearxiv.org
• Modular Manifolds - Thinking Machines Labthinkingmachines.ai
• Tensor-Transformer Variants are Surprisingly Performant — LessWronglesswrong.com
• Maybe I was too harsh on deep learning theory (three days ago) — LessWronglesswrong.com
• Do Machine Learning Models Memorize or Generalize?pair.withgoogle.com
• NL.pdfabehrouz.github.io
• Neural networks generalize because of this one weird trick — LessWronglesswrong.com
• The generalization phase diagram — LessWronglesswrong.com
• [2410.21265] Modular Duality in Deep Learningarxiv.org
• [2605.01172] A Theory of Generalization in Deep Learningarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org