There are exponentially many vectors with small inner product | Longest path search

lmao.bearblog.dev · 1,570 words · saved by 1 readers

An interesting observation that Alex stumbled across while reading some of Anthropic's work on LLM interpretability is the following Although it's only possible to have 𝑛 orthogonal vectors in an 𝑛 -dimensional space, it's possible to have exp ( 𝑛 ) many "almost orthogonal" ( 𝜀 cosine similarity) vectors in high-dimensional spaces. See the Johnson–Lindenstrauss lemma. While the JL lemma does indicate that this might be true (ish, in a certain sense) it is not obvious that this statement directly translates to the one given here. (I won't comment on the content on the rest of the linked post, some of which I find somewhat, uh, loose in construction, but I'll focus specifically on this point.) For example, a simple fact from linear algebra is that, given 𝑚 vectors 𝑥 1 , … , 𝑥 𝑚 ∈ 𝐑 𝑛 in 𝑛 dimensional space each of which has nonpositive inner product with any other vector 𝑥 𝑖 𝑇 𝑥 𝑗 ≤ 0 for 𝑖 ≠ 𝑗 , we have that 𝑚 ≤ 2 𝑛 . In other words: no more than 2 �

There are exponentially many vectors with small inner product – Longest path search There are exponentially many vectors with small inner product 09 Jul, 2025 An interesting observation that Alex stumbled across while reading some of Anthropic's work on LLM interpretability is the following Although it's only possible to have n orthogonal vectors in an n -dimensional space, it's possible to have exp ( n ) many "almost orthogonal" ( ε cosine similarity) vectors in high-dimensional spaces. See the Johnson–Lindenstrauss lemma . While the JL lemma does indicate that this mi

There are exponentially many vectors with small inner product | Longest path search

related reading