[2410.12101] The Persian Rug: solving toy models of superposition using large-scale symmetries

Abstract:We present a complete mechanistic description of the algorithm learned by a minimal non-linear sparse data autoencoder in the limit of large input dimension. The model, originally presented in arXiv:2209.10652, compresses sparse data vectors through a linear layer and decompresses using another linear layer followed by a ReLU activation. We notice that when the data is permutation symmetric (no input feature is privileged) large models reliably learn an algorithm that is sensitive to individual weights only through their large-scale statistics. For these models, the loss function becomes analytically tractable. Using this understanding, we give the explicit scalings of the loss at high sparsity, and show that the model is near-optimal among recently proposed architectures. In particular, changing or adding to the activation function any elementwise or filtering operation can at best improve the model's performance by a constant factor. Finally, we forward-engineer a model with the requisite symmetries and show that its loss precisely matches that of the trained models. Unlike the trained model weights, the low randomness in the artificial weights results in miraculous fractal structures resembling a Persian rug, to which the algorithm is oblivious. Our work contributes to neural network interpretability by introducing techniques for understanding the structure of autoencoders. Code to reproduce our results can be found at this https URL .

# link_587cls330t.pdf ## Metadata - PDFFormatVersion=1.5 - IsLinearized=false - IsAcroFormPresent=false - IsXFAPresent=false - IsCollectionPresent=false - IsSignaturesPresent=false - CreationDate=D:20241023011325Z - Creator=LaTeX with hyperref - ModDate=D:20241023011325Z - Custom.PTEX.Fullbanner=This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) kpathsea version 6.3.5 - Producer=pdfTeX-1.40.25 - Trapped=False ## Contents ### Page 1 The Persian Rug: Solving Toy Models of Superposition using Large-Scale SymmetriesTHE PERSIAN RUG: SOLVING TOY MODELS OF SUPER-POSITION USING LARGE-S

saved by

• Lucas Chen

related reading

• pdfopenreview.net
• [2503.02113] Deep Learning is Not So Mysterious or Differentarxiv.org
• [2604.21691] There Will Be a Scientific Theory of Deep Learningarxiv.org
• Toy Models of Superpositiontransformer-circuits.pub
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• [2608.27540] Towards a mathematical theory of superpositionarxiv.org
• [2602.15029] Symmetry in language statistics shapes the geometry of model representationsarxiv.org
• arxiv.org/pdf/2512.24880#page=3.56arxiv.org
• Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org
• Statistical Mechanics of Deep Learningganguli-gang.stanford.edu
• [2605.01172] A Theory of Generalization in Deep Learningarxiv.org