Transformer inference tricks - by Finbarr Timbers

How to make your model run faster than a greased pig

Special thanks to @cis_female for discussing the intricacies of sparsity with me, and @nostalgebraist for correcting an error in the quantization section; I now think that the evidence shows that quantizing, at least to 4 bits or more, has a very minimal tradeoff in terms of performance. I’m going to discuss a number of optimizations that can be done to make inference for transformers either faster or more efficient. By far the most common (and most important) optimization for a decoder is a KV cache. In a decoder model, the keys and values will be identical for the prompt for every…

related reading

• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Transformer inference tricks - by Finbarr Timbersartfintel.com
• All About Transformer Inferencejax-ml.github.io
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Fast Inference from Transformers via Speculative Decodingarxiv.org
• Efficient LLM inferencefinbarrtimbers.substack.com
• KV Caching Explained: Optimizing Transformer Inference Efficiencyhuggingface.co
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Fast Inference from Transformers via Speculative Decodingarxiv.org
• Fast Inference from Transformers via Speculative Decodingarxiv.org
• Speculative Decoding - philkravphilkrav.com
• Optimizing inference · Hugging Facehuggingface.co