vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention | vLLM Blog

Jun 20, 2023 • Woosuk Kwon*, Zhuohan Li*, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Yu, Joey Gonzalez, Hao Zhang, and Ion Stoica (* Equal Contribution) GitHub | Documentation | Paper LLMs promise to fundamentally change how we use AI across all industries. However, actually serving these models is challenging and can be surprisingly slow even on expensive hardware. Today we are excited to introduce vLLM, an open-source library for fast LLM inference and serving. vLLM utilizes PagedAttention, our new attention algorithm that effectively manages attention keys and values. vLLM equipped with PagedAttention redefines the new state of the art in LLM serving: it delivers up to 24x higher throughput than HuggingFace Transformers, without requiring any model architecture changes. vLLM has been developed at UC Berkeley and deployed at Chatbot Arena and Vicuna Demo for the past two months. It is the core technology that makes LLM serving affordable even for a small research team like LMSYS

Table of Contents GitHub | Documentation | Paper LLMs promise to fundamentally change how we use AI across all industries. However, actually serving these models is challenging and can be surprisingly slow even on expensive hardware. Today we are excited to introduce vLLM, an open-source library for fast LLM inference and serving. vLLM utilizes PagedAttention , our new attention algorithm that effectively manages attention keys and values. vLLM equipped with PagedAttention redefines the new state of the art in LLM serving: it delivers up to 24x higher throughput than HuggingFace Transformers,

saved by

• Mert Bozkır

related reading

• LLM Optimization via Synthetic Distillationanarchyai.substack.com
• Introduction to vLLM and PagedAttentionblog.runpod.io
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• Inside vLLM: Anatomy of a High-Throughput LLM Inference Systemvllm.ai
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Efficient Memory Management for Large Language Model Serving with PagedAttentionarxiv.org
• LLM Engineer's Almanac - Advisormodal.com
• Optimizing inference · Hugging Facehuggingface.co
• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• How LLM Inference Worksarpitbhayani.me
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com