Sparse Coding

Sparse coding is a class of unsupervised methods for learning sets of over-complete bases to represent data efficiently. The aim of sparse coding is to find a set of basis vectors ϕ i 𝜙 𝑖 such that we can represent an input vector x 𝑥 as a linear combination of these basis vectors: While techniques such as Principal Component Analysis (PCA) allow us to learn a complete set of basis vectors efficiently, we wish to learn an over-complete set of basis vectors to represent input vectors x∈ ℝ n 𝑥 ∈ 𝑅 𝑛 (i.e. such that k>n 𝑘 > 𝑛 ). The advantage of having an over-complete basis is that our basis vectors are better able to capture structures and patterns inherent in the input data. However, with an over-complete basis, the coefficients a i 𝑎 𝑖 are no longer uniquely determined by the input vector x 𝑥 . Therefore, in sparse coding, we introduce the additional criterion of sparsity to resolve the degeneracy introduced by over-completeness. Here, we define sparsity as having

Unsupervised Feature Learning and Deep Learning Tutorial UFLDL Tutorial Home !--> Sparse Coding 28 Aug 2013 ---> Sparse coding is a class of unsupervised methods for learning sets of over-complete bases to represent data efficiently. The aim of sparse coding is to find a set of basis vectors \mathbf{\phi}_i such that we can represent an input vector \mathbf{x} as a linear combination of these basis vectors: \begin{align} \mathbf{x} = \sum_{i=1}^k a_i \mathbf{\phi}_{i} \end{align} While techniques such as Principal Component Analysis (PCA) allow us to learn a complete set of basis vectors effic

related reading

• machine learning - Sparse coding vs. sparse PCA, are they the same thing? - Cross Validatedstats.stackexchange.com
• Toy Models of Superpositiontransformer-circuits.pub
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• Order Preserving Sparse Coding | IEEE Journals & Magazine | IEEE Xploreieeexplore.ieee.org
• [Interim research report] Taking features out of superposition with sparse autoencoders — LessWronglesswrong.com
• [2410.12101] The Persian Rug: solving toy models of superposition using large-scale symmetriesarxiv.org
• An Intuitive Explanation of Sparse Autoencoders for LLM Interpretability | Adam Karvonenadamkarvonen.github.io
• Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org
• Sparse PCA - Wikipediaen.wikipedia.org
• Overview of Sparse Modeling | Chan`s Jupytergoodboychan.github.io
• Compressed sensing - Wikipediaen.wikipedia.org
• 1 An Informal Introduction to Intelligencema-lab-berkeley.github.io