Speculative KV coding: losslessly compressing KV cache by up to ~4× using a predictor model

Lossless compression of a target model's KV cache by up to 4×, using a cheaper predictor model to drive an arithmetic coder.

Speculative KV coding: losslessly compressing KV cache by up to ~4× using a predictor model 8 May 2026 · 14 min read · Cover: Andreas Cellarius, Theoria Lunae , from Harmonia Macrocosmica (1660), engraved by Johannes van Loon. The size of LLM context grows by the day. KV caching is what makes running those long contexts affordable: it trades compute for memory so the model doesn’t re-prefill work it has already done. But as agentic workflows push contexts ever longer, storing and moving the cache starts to dominate everything. To get to the next order of magnitude of LLM capability, we need it

saved by

• Lailah Nabegu

related reading

• Compression and Intelligencegreene.sh
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Speculative Decoding - philkravphilkrav.com
• [2309.10668] Language Modeling Is Compressionarxiv.org
• Cross-Model KV Cache Transfer in LLM Families:A Closed-Form Linear Mapping for Prefill Reusearxiv.org
• Fast Inference from Transformers via Speculative Decodingarxiv.org
• KV Caching Explained: Optimizing Transformer Inference Efficiencyhuggingface.co
• Quantization from the ground upngrok.com
• The economics of speculative decoding | Doublewordblog.doubleword.ai
• [2412.14838] DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMsarxiv.org
• Transformer inference trickssubstack.com
• Speculative Decoding: How It Evolved, When It Stays Lossless, and What's Nextneurips2026-speculative-decoding.vercel.app