Efficient Memory Management for Large Language Model Serving with PagedAttention

N/A

Efficient Memory Management for Large Language Model Serving with PagedAttention Woosuk Kwon1,∗ Zhuohan Li1,∗ Siyuan Zhuang1 Ying Sheng1,2 Lianmin Zheng1 Cody Hao Yu3 Joseph E. Gonzalez1 Hao Zhang4 Ion Stoica1 1 UC Berkeley 2 Stanford University 3 Independent Researcher 4 UC San Diego…

related reading

• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• Introduction to vLLM and PagedAttentionblog.runpod.io
• vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention | vLLM Blogblog.vllm.ai
• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Inside vLLM: Anatomy of a High-Throughput LLM Inference Systemvllm.ai
• [2412.02252] Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarityarxiv.org
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• [2603.23516] MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokensarxiv.org
• The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
• [2412.14838] DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMsarxiv.org
• [2205.14135] FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenessarxiv.org