[2410.12101] The Persian Rug: solving toy models of superposition using large-scale symmetries

Abstract:We present a complete mechanistic description of the algorithm learned by a minimal non-linear sparse data autoencoder in the limit of large input dimension. The model, originally presented in arXiv:2209.10652, compresses sparse data vectors through a linear layer and decompresses using another linear layer followed by a ReLU activation. We notice that when the data is permutation symmetric (no input feature is privileged) large models reliably learn an algorithm that is sensitive to individual weights only through their large-scale statistics. For these models, the loss function becomes analytically tractable. Using this understanding, we give the explicit scalings of the loss at high sparsity, and show that the model is near-optimal among recently proposed architectures. In particular, changing or adding to the activation function any elementwise or filtering operation can at best improve the model's performance by a constant factor. Finally, we forward-engineer a model with the requisite symmetries and show that its loss precisely matches that of the trained models. Unlike the trained model weights, the low randomness in the artificial weights results in miraculous fractal structures resembling a Persian rug, to which the algorithm is oblivious. Our work contributes to neural network interpretability by introducing techniques for understanding the structure of autoencoders. Code to reproduce our results can be found at this https URL .

# link_587cls330t.pdf ## Metadata - PDFFormatVersion=1.5 - IsLinearized=false - IsAcroFormPresent=false - IsXFAPresent=false - IsCollectionPresent=false - IsSignaturesPresent=false - CreationDate=D:20241023011325Z - Creator=LaTeX with hyperref - ModDate=D:20241023011325Z - Custom.PTEX.Fullbanner=This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) kpathsea version 6.3.5 - Producer=pdfTeX-1.40.25 - Trapped=False ## Contents ### Page 1 The Persian Rug: Solving Toy Models of Superposition using Large-Scale SymmetriesTHE PERSIAN RUG: SOLVING TOY MODELS OF SUPER-POSITION USING LARGE-S

saved by

• Lucas Chen

related reading

• pdfopenreview.net
• Toy Models of Superpositiontransformer-circuits.pub
• [Interim research report] Taking features out of superposition with sparse autoencoders — LessWronglesswrong.com
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• An Intuitive Explanation of Sparse Autoencoders for LLM Interpretability | Adam Karvonenadamkarvonen.github.io
• Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org
• Do sparse autoencoders find "true features"? — LessWronglesswrong.com
• Matryoshka Sparse Autoencoders — LessWronglesswrong.com
• Autoencoder - Wikipediaen.wikipedia.org
• The Little Book of Deep Learningfleuret.org
• Interpretability with Sparse Autoencoders (Colab exercises) — LessWronglesswrong.com