Compression and Intelligence — Ryan Greene
greene.sh · 6,278 words · saved by 5 readers
On the relation between compression, intelligence, and LLMs.
Part One: Compression Pretraining Pretrained large language models are famously next-token predictors. The procedure of pretraining consists of randomly initializing a (typically Transformer) model θ\theta, taking a sequence of data X=(x1,x2,…,xT)X = (x_1, x_2, \dots, x_T), autoregressively predicting the probabilities of each symbol: pθ(xt∣x<t)p_\theta\left(x_t \mid x_{\lt t}\right), computing a loss ℓ\ell of how well the model predicted the true sequence XX, and backpropagating ℓ\ell to adjust the model’s weights to make XX more likely, with the hope that the absorbed regularities will…
saved by
related reading
- Compression is predictionngrok.com
- [2309.10668] Language Modeling Is Compressionarxiv.org
- LLMs can invent their own compression - Rajan Agarwalrajan.sh
- 0812.4360arxiv.org
- How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
- The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
- Data Compression Explainedmattmahoney.net
- Generalization Dynamics of LM Pre-training — Jiaxin Wenjiaxin-wen.github.io
- Can gzip be a language model?nathan.rs
- LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelsarxiv.org
- Composer2.pdfcursor.com
- Quantization from the ground upngrok.com