Speculative KV coding: losslessly compressing KV cache by up to ~4× using a predictor model

Lossless compression of a target model's KV cache by up to 4×, using a cheaper predictor model to drive an arithmetic coder.

Speculative KV coding: losslessly compressing KV cache by up to ~4× using a predictor model 8 May 2026 · 14 min read · Cover: Andreas Cellarius, Theoria Lunae , from Harmonia Macrocosmica (1660), engraved by Johannes van Loon. The size of LLM context grows by the day. KV caching is what makes running those long contexts affordable: it trades compute for memory so the model doesn’t re-prefill work it has already done. But as agentic workflows push contexts ever longer, storing and moving the cache starts to dominate everything. To get to the next order of magnitude of LLM capability, we need it

saved by

• Lailah Nabegu

related reading

• Can gzip be a language model?nathan.rs
• TurboQuant: Redefining AI efficiency with extreme compressionresearch.google
• Cache strategies · Hugging Facehuggingface.co
• [2309.10668] Language Modeling Is Compressionarxiv.org
• TurboQuant: Redefining AI efficiency with extreme compressionresearch.google
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• [2412.14838] DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMsarxiv.org
• KV Caching Explained: Optimizing Transformer Inference Efficiencyhuggingface.co
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• Composer2.pdfcursor.com
• LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelsarxiv.org
• Speculative Decoding - philkravphilkrav.com