vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention | vLLM Blog

Jun 20, 2023 • Woosuk Kwon*, Zhuohan Li*, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Yu, Joey Gonzalez, Hao Zhang, and Ion Stoica (* Equal Contribution) GitHub | Documentation | Paper LLMs promise to fundamentally change how we use AI across all industries. However, actually serving these models is challenging and can be surprisingly slow even on expensive hardware. Today we are excited to introduce vLLM, an open-source library for fast LLM inference and serving. vLLM utilizes PagedAttention, our new attention algorithm that effectively manages attention keys and values. vLLM equipped with PagedAttention redefines the new state of the art in LLM serving: it delivers up to 24x higher throughput than HuggingFace Transformers, without requiring any model architecture changes. vLLM has been developed at UC Berkeley and deployed at Chatbot Arena and Vicuna Demo for the past two months. It is the core technology that makes LLM serving affordable even for a small research team like LMSYS

Table of Contents GitHub | Documentation | Paper LLMs promise to fundamentally change how we use AI across all industries. However, actually serving these models is challenging and can be surprisingly slow even on expensive hardware. Today we are excited to introduce vLLM, an open-source library for fast LLM inference and serving. vLLM utilizes PagedAttention , our new attention algorithm that effectively manages attention keys and values. vLLM equipped with PagedAttention redefines the new state of the art in LLM serving: it delivers up to 24x higher throughput than HuggingFace Transformers,

saved by

• Mert Bozkır

related reading

• Introduction to vLLM and PagedAttentionblog.runpod.io
• Efficient Memory Management for Large Language Model Serving with PagedAttentionarxiv.org
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• Inside vLLM: Anatomy of a High-Throughput LLM Inference Systemvllm.ai
• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
• What We’ve Learned From A Year of Building with LLMs – Applied LLMsapplied-llms.org
• The State of LLM Serving in 2026: Ollama, SGLang, TensorRT, Triton, and vLLM | Canteenthecanteenapp.com
• Mosaic LLMs: GPT-3 quality formosaicml.com
• LLM Engineer's Almanac - Workloads | Modalmodal.com