Coherence — sparse-plex v2019.02
Finding out the spark of a dictionary 𝐷 is NP-hard since it involves considering combinatorially large number of selections of columns from 𝐷 . In this section we consider the coherence of a dictionary which is computationally tractable and quite useful in characterizing the solutions of sparse approximation problems. The coherence of a dictionary 𝐷 is defined as the maximum absolute inner product between two distinct atoms in the dictionary: If the dictionary consists of two orthonormal bases, then coherence is also known as mutual coherence or proximity. Since the atoms within each orthonormal basis are orthogonal to each other, the coherence is determined only by the inner products of atoms from one basis with another basis. We note that d ω i 𝑑 𝜔 𝑖 is the i 𝑖 -th column of synthesis matrix 𝐷 . Also H 𝐷 𝐻 𝐷 is the Gram matrix for 𝐷 whose elements are nothing but the inner-products of columns of 𝐷 . We note that by definition ‖ d ω ‖ 2
Coherence - sparse-plex v2019.02 --> Docs >> Sparse Signal Models >> Coherence Coherence ¶ Finding out the spark of a dictionary \(\DDD\) is NP-hard since it involves considering combinatorially large number of selections of columns from \(\DDD\) . In this section we consider the coherence of a dictionary which is computationally tractable and quite useful in characterizing the solutions of sparse approximation problems. Definition The coherence of a dictionary \(\DDD\) is defined as the maximum absolute inner product between two distinct atoms in the dictionary: \[\mu = \underset{j \neq k}{\t
Explore this link on the map →related reading
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
- https://www.deeplearningbook.org/contents/linear_algebra.htmldeeplearningbook.org
- Restricted isometry property - Wikipediaen.wikipedia.org
- Singular value decomposition - Wikipediaen.wikipedia.org
- Using spaced repetition systems to see through a piece of mathematicscognitivemedium.com
- Sparse matrix - Wikipediaen.wikipedia.org
- A short survey on almost orthogonal vectors in a few specific large dimensionsarxiv.org
- Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org
- 1404.5236 Sum-of-Squares Proofs and the Quest toward Optimal Algorithmsarxiv.org
- 1409.6366arxiv.org
- Sparse PCA - Wikipediaen.wikipedia.org
- [1305.5870] The Optimal Hard Threshold for Singular Values is 4/sqrt(3)arxiv.org