Speculative KV coding: losslessly compressing KV cache by up to ~4× using a predictor model

Lossless compression of a target model's KV cache by up to 4×, using a cheaper predictor model to drive an arithmetic coder.

Speculative KV coding: losslessly compressing KV cache by up to ~4× using a predictor model 8 May 2026 · 14 min read · Cover: Andreas Cellarius, Theoria Lunae , from Harmonia Macrocosmica (1660), engraved by Johannes van Loon. The size of LLM context grows by the day. KV caching is what makes running those long contexts affordable: it trades compute for memory so the model doesn’t re-prefill work it has already done. But as agentic workflows push contexts ever longer, storing and moving the cache starts to dominate everything. To get to the next order of magnitude of LLM capability, we need it

saved by

• Lailah Nabegu

related reading

• Compression and Intelligencegreene.sh
• TurboQuant: Redefining AI efficiency with extreme compressionresearch.google
• [2309.10668] Language Modeling Is Compressionarxiv.org
• Cache strategies · Hugging Facehuggingface.co
• TurboQuant: Redefining AI efficiency with extreme compressionresearch.google
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Cross-Model KV Cache Transfer in LLM Families:A Closed-Form Linear Mapping for Prefill Reusearxiv.org
• KV Caching Explained: Optimizing Transformer Inference Efficiencyhuggingface.co
• 2407.15891arxiv.org
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• Compression is predictionngrok.com
• [2412.14838] DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMsarxiv.org