LLMs can invent their own compression - Rajan Agarwal

rajan.sh · 2,504 words · saved by 5 readers

TL;DR: I co-trained a summarizer and a generator to learn a compression scheme for text in the same token space as the base model, so it can continue while staying close to full-context behavior on next-token prediction, using an order of magnitude fewer context tokens. Along the way the model discovers its own compression tricks: aggressive pruning, dense punctuation (lots of semicolons), and even occasionally switching into Mandarin to pack more information per token. We keep asking what kinds of problems LLMs could solve if you just let them think longer—hours, days, weeks. Recent work like OpenAI’s GPT-5.1-Codex-Max and the accelerating science experiments run a model for millions of output tokens on a single objective. If those tokens are all useful, that’s incredible for doing fundamental research and problem solving within engineering, math and science. I'm a strong believer that agents can work for a long time to solve hard problems. GPT-5.1-Codex-Max (...) is our first model n

LLMs can invent their own compression - Rajan Agarwal TL;DR: I co-trained a summarizer and a generator to learn a compression scheme for text in the same token space as the base model, so it can continue while staying close to full-context behavior on next-token prediction, using an order of magnitude fewer context tokens. Along the way the model discovers its own compression tricks: aggressive pruning, dense punctuation (lots of semicolons), and even occasionally switching into Mandarin to pack more information per token. State of Context Compaction We keep asking what kinds of problems LLMs

LLMs can invent their own compression - Rajan Agarwal

saved by

related reading