Compression and Intelligence — Ryan Greene

On the relation between compression, intelligence, and LLMs.

Part One: Compression Pretraining Pretrained large language models are famously next-token predictors. The procedure of pretraining consists of randomly initializing a (typically Transformer) model θ\theta, taking a sequence of data X=(x1,x2,…,xT)X = (x_1, x_2, \dots, x_T), autoregressively predicting the probabilities of each symbol: pθ(xt∣x<t)p_\theta\left(x_t \mid x_{\lt t}\right), computing a loss ℓ\ell of how well the model predicted the true sequence XX, and backpropagating ℓ\ell to adjust the model’s weights to make XX more likely, with the hope that the absorbed regularities will…

saved by

• Sophie Wang
• Brady Gho
• Julian H
• Jaeho Lee
• Evie Hu

related reading

• Compression is predictionngrok.com
• [2309.10668] Language Modeling Is Compressionarxiv.org
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• LLMs can invent their own compression - Rajan Agarwalrajan.sh
• Can gzip be a language model?nathan.rs
• Quantization from the ground upngrok.com
• Language Models are Elasticaclanthology.org
• [2509.14786] Pre-training under infinite computearxiv.org
• Reinventing Entropy | Compression is Intelligence Part 1youtube.com
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• The Intricate Link Between Compression and Predictionmindfulmodeler.substack.com
• arxiv.org/pdf/2505.24832arxiv.org