LLM inference optimization

Large language models (LLMs) have pushed text generation applications, such as chat and code completion models, to the next level by producing text that displays a high level of understanding and fluency. But what makes LLMs so powerful - namely their size - also presents challenges for inference. Basic inference is slow because LLMs have to be called repeatedly to generate the next token. The input sequence increases as generation progresses, which takes longer and longer for the LLM to process. LLMs also have billions of parameters, making it a challenge to store and handle all those weights in memory. This guide will show you how to use the optimization techniques available in Transformers to accelerate LLM inference. Hugging Face also provides Text Generation Inference (TGI), a library dedicated to deploying and serving highly optimized LLMs for inference. It includes more optimization features not included in Transformers, such as continuous batching for increasing throughput and

Optimizing inference · Hugging Face Transformers documentation Optimizing inference Transformers 🏡 View all docs AWS Trainium & Inferentia Accelerate Argilla AutoTrain Bitsandbytes CLI Chat UI Dataset viewer Datasets Deploying on AWS Diffusers Distilabel Evaluate Google Cloud Google TPUs Gradio Hub Hub Python Library Huggingface.js Inference Endpoints (dedicated) Inference Providers Kernels LeRobot Leaderboards Lighteval Microsoft Azure OpenEnv Optimum PEFT Reachy Mini Safetensors Sentence Transformers TRL Tasks Text Embeddings Inference Text Generation Inference Tokenizers Trackio Transforme

saved by

• Kevin Wang

related reading

• Accelerating Generative AI with PyTorch II: GPT, Fast – PyTorchpytorch.org
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• How to make LLMs go fastvgel.me
• Best practices to accelerate inference for large-scale production workloadstogether.ai
• Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• How LLM Inference Worksarpitbhayani.me
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Speculative Decoding - philkravphilkrav.com
• LLM Inference Economics from First Principlestensoreconomics.com
• Inference characteristics of Llama · Cursorcursor.com