Reinventing Entropy | Compression is Intelligence Part 1 - YouTube
youtube.com · 6,187 words · saved by 3 readers
Follow along using the transcript. Follow along using the transcript.
How to study the compressibility of language. Check out our virtual career fair: https://3b1b.co/talent See new projects before they go live: https://3b1b.co/support Animation credit: Manim scenes by Aaron Gostein and Grant Sanderson Shannon’s story, as well as those for various pi creatures, by Mitchell Zemil. Lunar robot and prediction/compression coin by Paul Dancstep NanoGPT animations by Clayton Rabideau The way of visualizing entropy shown here is something I first came across in this excellent post by Chris Olah: https://colah.github.io/posts/2015-09-Visual-Information/ Shannon’s…
saved by
related reading
- Visual Information Theory -- colah's blogcolah.github.io
- Compression and Intelligencegreene.sh
- Compression is predictionngrok.com
- [2206.07867] A visual introduction to information theoryarxiv.org
- LLMs can invent their own compression - Rajan Agarwalrajan.sh
- Why Information Theory is Important - Computerphileyoutube.com
- Six (and a half) intuitions for KL divergence — LessWronglesswrong.com
- Information entropy (Foundations of information theory: Part 2) - Matthew N. Bernsteinmbernste.github.io
- Introduction to abstract entropy — LessWronglesswrong.com
- [2309.10668] Language Modeling Is Compressionarxiv.org
- How might LLMs store facts | Deep Learning Chapter 7youtube.com
- But what are Hamming codes? The origin of error correctionyoutube.com