LLM inference optimization

Large language models (LLMs) have pushed text generation applications, such as chat and code completion models, to the next level by producing text that displays a high level of understanding and fluency. But what makes LLMs so powerful - namely their size - also presents challenges for inference. Basic inference is slow because LLMs have to be called repeatedly to generate the next token. The input sequence increases as generation progresses, which takes longer and longer for the LLM to process. LLMs also have billions of parameters, making it a challenge to store and handle all those weights in memory. This guide will show you how to use the optimization techniques available in Transformers to accelerate LLM inference. Hugging Face also provides Text Generation Inference (TGI), a library dedicated to deploying and serving highly optimized LLMs for inference. It includes more optimization features not included in Transformers, such as continuous batching for increasing throughput and

Optimizing inference · Hugging Face Transformers documentation Optimizing inference Transformers 🏡 View all docs AWS Trainium & Inferentia Accelerate Argilla AutoTrain Bitsandbytes CLI Chat UI Dataset viewer Datasets Deploying on AWS Diffusers Distilabel Evaluate Google Cloud Google TPUs Gradio Hub Hub Python Library Huggingface.js Inference Endpoints (dedicated) Inference Providers Kernels LeRobot Leaderboards Lighteval Microsoft Azure OpenEnv Optimum PEFT Reachy Mini Safetensors Sentence Transformers TRL Tasks Text Embeddings Inference Text Generation Inference Tokenizers Trackio Transforme

saved by

• Kevin Wang

related reading

• Accelerating Generative AI with PyTorch II: GPT, Fast – PyTorchpytorch.org
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Looking back at speculative decodingresearch.google
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• How to make LLMs go fastvgel.me
• Efficient LLM inferencefinbarrtimbers.substack.com
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• Best practices to accelerate inference for large-scale production workloadstogether.ai
• Speculative Decoding - philkravphilkrav.com
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• How LLM Inference Worksarpitbhayani.me
• Fast Inference from Transformers via Speculative Decodingarxiv.org