A tale of three theories: sparsity, frustration, and statistical field theory — LessWrong

This post is an informal preliminary writeup of a project that I've been working on with friends and collaborators. Some of the theory was developed…

x A tale of three theories: sparsity, frustration, and statistical field theory — LessWrong AI Frontpage 63 A tale of three theories: sparsity, frustration, and statistical field theory by Dmitry Vaintrob 25th Jan 2026 22 min read 0 63 This post is an informal preliminary writeup of a project that I've been working on with friends and collaborators. Some of the theory was developed jointly with Zohar Ringel, and we hope to write a more formal paper on it this year. Experiments are joint with Lucas Teixeira (and also an extensive use of llm assistants). This work is part of the research agenda

related reading

• Toy Models of Superpositiontransformer-circuits.pub
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• Statistical Mechanics of Deep Learningganguli-gang.stanford.edu
• On neural scaling and the quanta hypothesisericjmichaud.com
• Interpretability Dreamstransformer-circuits.pub
• [2604.21691] There Will Be a Scientific Theory of Deep Learningarxiv.org
• [2410.12101] The Persian Rug: solving toy models of superposition using large-scale symmetriesarxiv.org
• What Would Non-Linear Features Actually Look Like? — Liv Gortonlivgorton.com
• A Theory of Deep Learning | Elements of a Vector Spaceelonlit.com
• [Interim research report] Taking features out of superposition with sparse autoencoders — LessWronglesswrong.com
• Growth and Form in a Toy Model of Superposition — LessWronglesswrong.com
• Maybe I was too harsh on deep learning theory (three days ago) — LessWronglesswrong.com