[2412.02252] Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity

Abstract:The increasing context window size in Large Language Models (LLMs), such as the GPT and LLaMA series, has improved their ability to tackle complex, long-text tasks, but at the cost of inference efficiency, particularly regarding memory and computational complexity. Existing methods, including selective token retention and window-based attention, improve efficiency but risk discarding important tokens needed for future text generation. In this paper, we propose an approach that enhances LLM efficiency without token loss by reducing the memory and computational load of less important tokens, rather than discarding this http URL address two challenges: 1) investigating the distribution of important tokens in the context, discovering recent tokens are more important than distant tokens in context, and 2) optimizing resources for distant tokens by sharing attention scores across layers. The experiments show that our method saves $35\%$ KV cache without compromising the performance.

Abstract:The increasing context window size in Large Language Models (LLMs), such as the GPT and LLaMA series, has improved their ability to tackle complex, long-text tasks, but at the cost of inference efficiency, particularly regarding memory and computational complexity. Existing methods, including selective token retention and window-based attention, improve efficiency but risk discarding important tokens needed for future text generation. In this paper, we propose an approach that enhances LLM efficiency without token loss by reducing the memory and computational load of less important to

related reading

• [2412.14838] DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMsarxiv.org
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• [2506.06266] Cartridges: Lightweight and general-purpose long context representations via self-studyarxiv.org
• Reimagining LLM Memory: Using Context as Training Data Unlocks Models That Learn at Test-Time | NVIDIA Technical Blogdeveloper.nvidia.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Mediumblog.gopenai.com
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• KV Caching Explained: Optimizing Transformer Inference Efficiencyhuggingface.co
• Cache strategies · Hugging Facehuggingface.co
• 2502.11089arxiv.org
• Subquadratic — How SSA Makes Long Context Practicalsubq.ai