Transformer inference tricks - by Finbarr Timbers

How to make your model run faster than a greased pig

Special thanks to @cis_female for discussing the intricacies of sparsity with me, and @nostalgebraist for correcting an error in the quantization section; I now think that the evidence shows that quantizing, at least to 4 bits or more, has a very minimal tradeoff in terms of performance. I’m going to discuss a number of optimizations that can be done to make inference for transformers either faster or more efficient. By far the most common (and most important) optimization for a decoder is a KV cache. In a decoder model, the keys and values will be identical for the prompt for every…

related reading

• Transformer inference tricks - by Finbarr Timbersartfintel.com
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• All About Transformer Inferencejax-ml.github.io
• Fast Inference from Transformers via Speculative Decodingarxiv.org
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Fast Inference from Transformers via Speculative Decodingarxiv.org
• Optimizing inference · Hugging Facehuggingface.co
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• KV Caching Explained: Optimizing Transformer Inference Efficiencyhuggingface.co
• How to make LLMs go fastvgel.me