Stanford CRFM

Large language models (LLM) such as ChatGPT or Llama have received unprecedented attention lately. However, they remain massively expensive to run. Even though generating a single response can cost about $0.01 (a few seconds of an 8xA100 instance on AWS), the costs quickly add up when scaling to billions of users, who could have multiple daily interactions with such LLMs. Some use cases are more expensive, like code auto-completion, because it runs whenever a new character is typed. As LLM applications multiply, even small efficiency gains to the generation time can have a massive impact. LLM inference (or “decoding”) is an iterative process: tokens are generated one at a time. Generating full sentences of N tokens requires N forward passes through the model. Fortunately, it is possible to cache previously calculated tokens: this means that a single generation step does not depend on the context length, except for a single operation, the attention. This operation does not scale well wi

Stanford CRFM Flash-Decoding for long-context inference Authors: Tri Dao and Daniel Haziza and Francisco Massa and Grigory Sizov Motivation Large language models (LLM) such as ChatGPT or Llama have received unprecedented attention lately. However, they remain massively expensive to run. Even though generating a single response can cost about $0.01 (a few seconds of an 8xA100 instance on AWS), the costs quickly add up when scaling to billions of users, who could have multiple daily interactions with such LLMs. Some use cases are more expensive, like code auto-completion, because it runs wheneve

related reading

• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Optimizing inference · Hugging Facehuggingface.co
• Recursive Language Models | Alex L. Zhangalexzhang13.github.io
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Speculative Decoding - philkravphilkrav.com
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• Inference characteristics of Llama · Cursorcursor.com
• Subquadratic — How SSA Makes Long Context Practicalsubq.ai
• Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
• Bits, FLOPS, and Watts: A Systems-Level Perspective of Scaling LLMs — Part 1 | by Asheesh Goja | Mediummedium.com