Prompt Cache: Modular Attention Reuse for Low-Latency Inference

N/A

P ROMPT C ACHE : M ODULAR ATTENTION R EUSE FOR L OW-L ATENCY I NFERENCE In Gim 1 Guojun Chen 1 Seung-seob Lee 1 Nikhil Sarda 2 Anurag Khandelwal 1 Lin Zhong 1 A BSTRACT We present Prompt Cache, an approach for accelerating inference for large language models (LLM) by reusing arXiv:2311.04934v2 [cs.CL] 25 Apr 2024…

related reading

• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• [2205.14135] FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenessarxiv.org
• 2502.11089arxiv.org
• KV Caching Explained: Optimizing Transformer Inference Efficiencyhuggingface.co
• [2506.06266] Cartridges: Lightweight and general-purpose long context representations via self-studyarxiv.org
• [2412.02252] Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarityarxiv.org
• Efficient Memory Management for Large Language Model Serving with PagedAttentionarxiv.org
• All About Transformer Inferencejax-ml.github.io
• [2304.13343] Unleashing Infinite-Length Input Capacity for Large-scale Language Models with Self-Controlled Memory Systemarxiv.org
• Continuous batching from first principleshuggingface.co