Efficient LLM inference - by Finbarr Timbers
finbarrtimbers.substack.com · 2,208 words · saved by 3 readers
On quantization, distillation, and efficiency
Lately, I’ve been thinking a lot about inference, and particularly, how to serve a given LLM more efficiently. The scenario is as follows: Your boss comes to you and says Hey Finbarr, we’re about to go bankrupt because we’re spending all of our investor’s money on GPUs serving our 300B parameter model that raps in the style of John Kenneth Galbraith. What can we do? Broadly speaking, there are three main classes of things you can do: You can quantize the parameters of your model (quantization), where you keep your model exactly the same, but use less precision for each of the parameters.…
saved by
related reading
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Quantization from the ground upngrok.com
- Optimizing inference · Hugging Facehuggingface.co
- A Guide to Quantization in LLMs | Symbl.aisymbl.ai
- On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
- LLM Optimization via Synthetic Distillationanarchyai.substack.com
- 2305.14314arxiv.org
- Best practices to accelerate inference for large-scale production workloadstogether.ai
- How is LLaMa.cpp possible?finbarr.ca
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com