Zipfian grokking | Jasper Gilley

A toy problem where wide abstractions fight deep ones — and a test bed for measuring the gap between dataset MDL and data-generating process MDL.

Zipfian grokking | Jasper Gilley ← All Posts Zipfian grokking Jasper Gilley TLDR: we modify the data distribution of a typical modular addition grokking setup to follow Zipf's Law, and show that it leads to persistent, predictable instability in grokked models' behavior. The imbalanced distribution causes the model to oscillate between a more generalizing and a more memorizing solution. We believe that this is a promising toy problem for testing new methods of eliciting generalization. One of the reasons for the successes of the scaling era of AI has been passive regularization: throwing

saved by

• Akira Yoshiyama
• Tommy He

related reading

• [2608.13335] Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Lawsarxiv.org
• [1911.08731] Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationarxiv.org
• Do Machine Learning Models Memorize or Generalize?pair.withgoogle.com
• [2605.01172] A Theory of Generalization in Deep Learningarxiv.org
• Alexi Gladstone | Explorative Modeling -- Unlocking a Third Pretraining Axis and End-to-End Generationalexiglad.github.io
• GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
• [2201.02177] Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetsarxiv.org
• Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetsmathai-iclr.github.io
• Scaling Laws, Carefully | Lil'Loglilianweng.github.io
• A Theory of Deep Learning | Elements of a Vector Spaceelonlit.com
• Towards Understanding Grokking: An Effective Theory of Representation Learningarxiv.org
• Lil'Loglilianweng.github.io