Can gzip be a language model?
A while back I wrote about language modeling without neural networks, where I generated Shakespeare with an unbounded n-gram model: no weights, no training, …
A while back I wrote about language modeling without neural networks , where I generated Shakespeare with an unbounded n-gram model: no weights, no training, just counting. Fortuitously, I came across the paper Language Modeling is Compression , which mentioned the compression–prediction equivalence : every prediction model is inherently a compressor, and all compression algorithms are prediction models . This led to the natural question: can gzip do language modeling? 1 No neural network, no learned parameters, nothing. Just the compressor that ships with your operating system. You prime it w
Explore this link on the map →saved by
related reading
- Language Modeling Without Neural Networksnathan.rs
- [2309.10668] Language Modeling Is Compressionarxiv.org
- ChatGPT Is a Blurry JPEG of the Web | The New Yorkernewyorker.com
- Language Modelinglena-voita.github.io
- LLMs can invent their own compression - Rajan Agarwalrajan.sh
- What Is ChatGPT Doing … and Why Does It Work?-Stephen Wolfram Writingswritings.stephenwolfram.com
- The Intricate Link Between Compression and Predictionmindfulmodeler.substack.com
- Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
- LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelsarxiv.org
- Introduction to Large Language Models | Machine Learning | Google for Developersdevelopers.google.com
- Data Compression Explainedmattmahoney.net
- Generalized Language Models | Lil'Loglilianweng.github.io