Overview of Sparse Modeling | Chan`s Jupyter

In this post, it will be explained about what the sparse modeling is and why this algorithm is used. This post is the summary of “Mathematical principles in Machine Learning” offered from UNIST

Overview of Sparse Modeling Sparse modeling Problem of Sparse Modeling Summary Overview of Sparse Modeling Sparse modeling The definition of sparse is come from the distribution. Sparse means something that is small number or amount and spread out over an area(or distribution). For example, sparse vectors and matrix have most of zeros and only a few number of non-zero valued elements. So why do we bring the concept of sparse to Machine Learning? Sparse modeling has advantages from training model. For example, consider about simple linear regression. y = θ 0 + θ 1 t + θ 2 t 2 + ⋯ + θ n t n y =

related reading

• Toy Models of Superpositiontransformer-circuits.pub
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• An Intuitive Explanation of Sparse Autoencoders for LLM Interpretability | Adam Karvonenadamkarvonen.github.io
• Regularization (mathematics) - Wikipediaen.wikipedia.org
• Sparsity Bluesbetanalpha.github.io
• Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org
• Sparse Attention Post-Training for Mechanistic Interpretabilityarxiv.org
• Unsupervised Feature Learning and Deep Learning Tutorialufldl.stanford.edu
• Less can be More: Sparsity as a Paradigm for LLM Developmentluxcapital.com
• L1 regularization: sparsity through singularitiesejenner.com
• What Is Ridge Regression? | IBMibm.com
• Weight-Sparse Circuits May Be Interpretable Yet Unfaithful — LessWronglesswrong.com