Efficient LLM inference - by Finbarr Timbers
finbarrtimbers.substack.com · 2,208 words · saved by 3 readers
On quantization, distillation, and efficiency
Lately, I’ve been thinking a lot about inference, and particularly, how to serve a given LLM more efficiently. The scenario is as follows: Your boss comes to you and says Hey Finbarr, we’re about to go bankrupt because we’re spending all of our investor’s money on GPUs serving our 300B parameter model that raps in the style of John Kenneth Galbraith. What can we do? Broadly speaking, there are three main classes of things you can do: You can quantize the parameters of your model (quantization), where you keep your model exactly the same, but use less precision for each of the parameters.…
saved by
related reading
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
- Quantization from the ground upngrok.com
- How To Scale Your Modeljax-ml.github.io
- How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
- Transformer inference trickssubstack.com
- A Guide to Quantization in LLMs | Symbl.aisymbl.ai
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- Spending Inference Time - Kevin Lukevinlu.ai
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- All About Transformer Inferencejax-ml.github.io