Introduction to vLLM and PagedAttention

vLLM is an open-source LLM inference and serving engine that utilizes a novel memory allocation algorithm called PagedAttention. It can run your models with up to 24x higher throughput than HuggingFace Transformers (HF) and up to 3.5x higher throughput than HuggingFace Text Generation Inference (TGI). How does vLLM achieve this throughput improvement? Existing systems waste 60%-80% of the KV-Cache (LLM memory), whereas vLLM achieves near-optimal memory usage with a mere waste of under 4%. Because of this improved memory usage, we require fewer GPUs to achieve the same output, so the throughput is significantly higher than that of other inference engines. Thousands of companies use vLLM to reduce inference costs. For example, LMSYS, the organization behind Chatbot Arena and Vicuna, cut the number of GPUs used to serve their growing traffic of ~45k daily requests by 50% while serving 2-3x more requests per second. vLLM leverages PagedAttention to reduce memory waste and achieve these eff

What is vLLM? vLLM is an open-source LLM inference and serving engine that utilizes a novel memory allocation algorithm called PagedAttention. It can run your models with up to 24x higher throughput than HuggingFace Transformers (HF) and up to 3.5x higher throughput than HuggingFace Text Generation Inference (TGI). How does vLLM achieve this throughput improvement? Existing systems waste 60%-80% of the KV-Cache (LLM memory), whereas vLLM achieves near-optimal memory usage with a mere waste of under 4%. Because of this improved memory usage, we require fewer GPUs to achieve the same output,…

saved by

• Tomi Jaga

related reading

• vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention | vLLM Blogblog.vllm.ai
• Inside vLLM: Anatomy of a High-Throughput LLM Inference Systemvllm.ai
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Efficient Memory Management for Large Language Model Serving with PagedAttentionarxiv.org
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• Optimizing inference · Hugging Facehuggingface.co
• Model Runner V2: A Modular and Faster Core for vLLM | vLLM Blogvllm.ai
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• How LLM Inference Worksarpitbhayani.me
• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
• LLM Engineer's Almanac - Advisormodal.com