How to make LLMs go fast

In my last post, we made a transformer by hand. There, we used the classic autoregressive sampler, along the lines of: This approach to inference is elegant and cuts to the heart of how LLMs work—they're autoregressive, consuming their own output. And for our toy model with merely thousands of parameters, it worked completely fine. Unfortunately, for real models it's far too slow1. Why is that, and how can we make it faster? This post is a long and wide-ranging survey of a bunch of different ways to make LLMs go brrrr, from better hardware utilization to clever decoding tricks. It's not completely exhaustive, and isn't the most in-depth treatment of every topic—I'm not an expert on all these things! But hopefully you'll find the information here a useful jumping off point to learn more about the topics you're interested in. (I tried to include links to relevant papers and blog posts where applicable.) There are two main reasons that inference with the plain autoregressive generate func

How to make LLMs go fast Posted December 18, 2023 In my last post , we made a transformer by hand. There, we used the classic autoregressive sampler, along the lines of: def generate ( prompt : str , tokens_to_generate : int ) -> str : tokens = tokenize(prompt) for i in range (tokens_to_generate): next_token = model(tokens) tokens.append(next_token) return detokenize(tokens) This approach to inference is elegant and cuts to the heart of how LLMs work—they're autoregressive , consuming their own output. And for our toy model with merely thousands of parameters, it worked completely fine. Unfort

related reading

• Optimizing inference · Hugging Facehuggingface.co
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• How LLM Inference Worksarpitbhayani.me
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Speculative Decoding - philkravphilkrav.com
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Fast Inference from Transformers via Speculative Decodingarxiv.org
• Best practices to accelerate inference for large-scale production workloadstogether.ai
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com