flâneur

Beyond Kolmogorov and Shannon — LessWrong

lesswrong.com · 3,770 words · saved by 1 readers

This post is the first in a sequence that will describe James Crutchfield's Computational Mechanics framework. We feel this is one of the most theoretically sound and promising approaches towards understanding Transformers in particular and interpretability more generally. As a heads up: Crutchfield's framework will take many posts to fully go through, but even if you don't make it all the way through there are still many deep insights we hope you will pick up along the way. EDIT: since there was some confusion about this in the comments: These initial posts are supposed to be an introductionary and won't get into the actually novel aspects of Crutchfield's framework yet. It's also not a dunk on existing information- theoretic measures - rather an ode! To better understand the capability and limitations of large language models it is crucial to understand the inherent structure and uncertainty ('entropy') of language data. It is natural to quantify this structure with complexity measu

x Beyond Kolmogorov and Shannon — LessWrong Kolmogorov Complexity Agency World Modeling AI Frontpage 63 Beyond Kolmogorov and Shannon by Alexander Gietelink Oldenziel , Adam Shai 25th Oct 2022 AI Alignment Forum 6 min read 22 63 Ω 16 This post is the first in a sequence that will describe James Crutchfield's Computational Mechanics framework. We feel this is one of the most theoretically sound and promising approaches towards understanding Transformers in particular and interpretability more generally. As a heads up: Crutchfield's framework will take many posts to fully go through, but even if

related reading