LLMs can invent their own compression - Rajan Agarwal

TL;DR: I co-trained a summarizer and a generator to learn a compression scheme for text in the same token space as the base model, so it can continue while staying close to full-context behavior on next-token prediction, using an order of magnitude fewer context tokens. Along the way the model discovers its own compression tricks: aggressive pruning, dense punctuation (lots of semicolons), and even occasionally switching into Mandarin to pack more information per token. We keep asking what kinds of problems LLMs could solve if you just let them think longer—hours, days, weeks. Recent work like OpenAI’s GPT-5.1-Codex-Max and the accelerating science experiments run a model for millions of output tokens on a single objective. If those tokens are all useful, that’s incredible for doing fundamental research and problem solving within engineering, math and science. I'm a strong believer that agents can work for a long time to solve hard problems. GPT-5.1-Codex-Max (...) is our first model n

LLMs can invent their own compression - Rajan Agarwal TL;DR: I co-trained a summarizer and a generator to learn a compression scheme for text in the same token space as the base model, so it can continue while staying close to full-context behavior on next-token prediction, using an order of magnitude fewer context tokens. Along the way the model discovers its own compression tricks: aggressive pruning, dense punctuation (lots of semicolons), and even occasionally switching into Mandarin to pack more information per token. State of Context Compaction We keep asking what kinds of problems LLMs

saved by

• Curtis Chong

related reading

• LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelsarxiv.org
• [2309.10668] Language Modeling Is Compressionarxiv.org
• Can gzip be a language model?nathan.rs
• Composer2.pdfcursor.com
• Training Composer for longer horizons · Cursorcursor.com
• GenAI Handbookgenai-handbook.github.io
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• LongLLMLingua Prompt Compression Guide | LlamaIndexblog.llamaindex.ai
• Recursive Language Models | Alex L. Zhangalexzhang13.github.io
• [2506.06266] Cartridges: Lightweight and general-purpose long context representations via self-studyarxiv.org
• On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io