Transformer inference tricks - by Finbarr Timbers

I’m going to discuss a number of optimizations that can be done to make inference for transformers either faster or more efficient. Artificial Fintelligence is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. By far the most common (and most important) optimization for a decoder is a KV cache. In a decoder model, the keys and values will be identical for the prompt for every iteration of decoding. Moreover, once you’ve ran a token through, the keys and values will be the same for that token for every subsequent iteration. As a result, you can cache the prompt, and incrementally add the KV tensors for each token to the cache as they are decoded. Doing so removes a lot of compute. Inside the attention mechanism, we’re able to go from multiplying two tensors of shape (batch, context_length, feature_dim) to multiplying a query tensor of shape (batch, 1, feature_dim) with your KV tensors of shape (batch, context_length, f

Transformer inference tricks How to make your model run faster than a greased pig Finbarr Timbers Nov 23, 2023 33 7 5 Share Transformer inference tricks Special thanks to @cis_female for discussing the intricacies of sparsity with me, and @nostalgebraist for correcting an error in the quantization section; I now think that the evidence shows that quantizing, at least to 4 bits or more, has a very minimal tradeoff in terms of performance. I’m going to discuss a number of optimizations that can be done to make inference for transformers either faster or more efficient. Artificial Fintelligence i

related reading

• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Fast Inference from Transformers via Speculative Decodingarxiv.org
• Optimizing inference · Hugging Facehuggingface.co
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Speculative Decoding - philkravphilkrav.com
• Best practices to accelerate inference for large-scale production workloadstogether.ai
• How to make LLMs go fastvgel.me
• A Guide to AI Inference Engineering - ByteByteGo Newsletterblog.bytebytego.com
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• [2207.00032] DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalearxiv.org
• Accelerated Inference for Large Transformer Models Using NVIDIA Triton Inference Server | NVIDIA Technical Blogdeveloper.nvidia.com