Compression is prediction | ngrok blog
Compression and LLMs are trying to solve the exact same problem: predicting what comes next. Learn the fundamentals of compression and how better prediction leads to better shrinkage.
I was reading about compression recently when I stumbled upon something crazy: that compressors and LLMs are, at their core, trying to solve the exact same problem. In this post, I’m going to walk us through the basics of compression to understand its deep relationship with language modeling. It’s probably going to blow your mind. How compression works There are many ways of shrinking data. Take minification, for example: it works by stripping code down to the bare minimum that machines need to parse. Human-readable variables are reduced to single letters; whitespace and comments are…
saved by
related reading
- Compression and Intelligencegreene.sh
- [2309.10668] Language Modeling Is Compressionarxiv.org
- Data Compression Explainedmattmahoney.net
- LLMs can invent their own compression - Rajan Agarwalrajan.sh
- Can gzip be a language model?nathan.rs
- LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelsarxiv.org
- ChatGPT Is a Blurry JPEG of the Web | The New Yorkernewyorker.com
- Lossless compressionen.wikipedia.org
- Entropy coding - Wikipediaen.wikipedia.org
- 0812.4360arxiv.org
- 500'000€ Prize for Compressing Human Knowledgeprize.hutter1.net
- The Intricate Link Between Compression and Predictionmindfulmodeler.substack.com