vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention | vLLM Blog

Jun 20, 2023 • Woosuk Kwon*, Zhuohan Li*, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Yu, Joey Gonzalez, Hao Zhang, and Ion Stoica (* Equal Contribution) GitHub | Documentation | Paper LLMs promise to fundamentally change how we use AI across all industries. However, actually serving these models is challenging and can be surprisingly slow even on expensive hardware. Today we are excited to introduce vLLM, an open-source library for fast LLM inference and serving. vLLM utilizes PagedAttention, our new attention algorithm that effectively manages attention keys and values. vLLM equipped with PagedAttention redefines the new state of the art in LLM serving: it delivers up to 24x higher throughput than HuggingFace Transformers, without requiring any model architecture changes. vLLM has been developed at UC Berkeley and deployed at Chatbot Arena and Vicuna Demo for the past two months. It is the core technology that makes LLM serving affordable even for a small research team like LMSYS

Table of Contents GitHub | Documentation | Paper LLMs promise to fundamentally change how we use AI across all industries. However, actually serving these models is challenging and can be surprisingly slow even on expensive hardware. Today we are excited to introduce vLLM, an open-source library for fast LLM inference and serving. vLLM utilizes PagedAttention , our new attention algorithm that effectively manages attention keys and values. vLLM equipped with PagedAttention redefines the new state of the art in LLM serving: it delivers up to 24x higher throughput than HuggingFace Transformers,

saved by

• Mert Bozkır

related reading

• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
• Optimizing inference · Hugging Facehuggingface.co
• How to make LLMs go fastvgel.me
• Optimized Inference Deployment · Hugging Facehuggingface.co
• How LLM Inference Worksarpitbhayani.me
• The State of LLM Serving in 2026: Ollama, SGLang, TensorRT, Triton, and vLLM | Canteenthecanteenapp.com
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• Patterns for Building LLM-based Systems & Productseugeneyan.com
• Model Runner V2: A Modular and Faster Core for vLLM | vLLM Blogvllm.ai
• Inference characteristics of Llama · Cursorcursor.com