[2412.02252] Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity

Abstract:The increasing context window size in Large Language Models (LLMs), such as the GPT and LLaMA series, has improved their ability to tackle complex, long-text tasks, but at the cost of inference efficiency, particularly regarding memory and computational complexity. Existing methods, including selective token retention and window-based attention, improve efficiency but risk discarding important tokens needed for future text generation. In this paper, we propose an approach that enhances LLM efficiency without token loss by reducing the memory and computational load of less important tokens, rather than discarding this http URL address two challenges: 1) investigating the distribution of important tokens in the context, discovering recent tokens are more important than distant tokens in context, and 2) optimizing resources for distant tokens by sharing attention scores across layers. The experiments show that our method saves $35\%$ KV cache without compromising the performance.

Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity Da Ma1 , Lu Chen1* , Situo Zhang1 , Yuxun Miao1 , Su Zhu2 Zhi Chen2 Hongshen Xu1 Hanqi Li1 , Shuai Fan3 , Lei Pan3 , Kai Yu1* 1 X-LANCE Lab, Department of Computer Science…

related reading

• [2412.14838] DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMsarxiv.org
• 2407.15891arxiv.org
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencearxiv.org
• [2506.06266] Cartridges: Lightweight and general-purpose long context representations via self-studyarxiv.org
• LLMs can invent their own compression - Rajan Agarwalrajan.sh
• KV Caching Explained: Optimizing Transformer Inference Efficiencyhuggingface.co
• The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
• Reimagining LLM Memory: Using Context as Training Data Unlocks Models That Learn at Test-Time | NVIDIA Technical Blogdeveloper.nvidia.com
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Context Parallelism for Scalable Million-Token Inferencearxiv.org
• 2306.14048arxiv.org