Compression is prediction | ngrok blog
Compression and LLMs are trying to solve the exact same problem: predicting what comes next. Learn the fundamentals of compression and how better prediction leads to better shrinkage.
I was reading about compression recently when I stumbled upon something crazy: that compressors and LLMs are, at their core, trying to solve the exact same problem. In this post, I’m going to walk us through the basics of compression to understand its deep relationship with language modeling. It’s probably going to blow your mind. How compression works There are many ways of shrinking data. Take minification, for example: it works by stripping code down to the bare minimum that machines need to parse. Human-readable variables are reduced to single letters; whitespace and comments are…
saved by
related reading
- Compression and Intelligencegreene.sh
- Can gzip be a language model?nathan.rs
- [2309.10668] Language Modeling Is Compressionarxiv.org
- LLMs can invent their own compression - Rajan Agarwalrajan.sh
- Reinventing Entropy | Compression is Intelligence Part 1youtube.com
- The Intricate Link Between Compression and Predictionmindfulmodeler.substack.com
- Quantization from the ground upngrok.com
- ChatGPT Is a Blurry JPEG of the Web | The New Yorkernewyorker.com
- How Lossless Data Compression Works | Quanta Magazinequantamagazine.org
- How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
- How does image compression work? | Lee Robinsonleerob.com
- Data Compression Explainedmattmahoney.net