✳flâneur — a map of the web's best reading

Some ML-Related Math I Now Understand Better — LessWrong

lesswrong.com · 2,797 words · saved by 1 readers

Let's say I want to understand what a weight matrix does. A large table of numbers isn't really helpful, I want something better. Here is a linear transformation which is much easier to understand than a table of numbers: f ( x ) = λ ⟨ v , x ⟩ u (where u and v are unit vectors, λ is a non-negative scalar, and ⟨ v , x ⟩ is the dot product between v and x ). f is reading in the v direction, and writing in the u direction with a magnitude scaled by λ . I can understand this much more clearly and do nice interpretability with it. Therefore, if I know things about my embedding space (for example, using the logit lens), I can tell what f is doing. Sadly, not all linear transformations can be expressed in this way: it's just a rank-1 transformation, so there is no hope of capturing something as complex as a usual linear transformation. But what if I allow myself to sum many simple transformations? Then I could just look at each operation independently. More precisely, he

x Some ML-Related Math I Now Understand Better — LessWrong Logic & Mathematics AI Frontpage 50 Some ML-Related Math I Now Understand Better by Fabien Roger 9th Mar 2023 5 min read 6 50 Here are some simple Math facts rarely taught in ML & Math lectures: SVD is decomposing a matrix into a sum of simple read-and-write operations There is exponentially much room for close vectors in high dimensional space Layer Normalization is a projection SVD Is Decomposing a Matrix Into a Sum of Simple Read and Write Operations Thanks to zfurman, on EleutherAI, which told me about the core idea. Let's say I wa

Explore this link on the map →

related reading