Transformer Inference Arithmetic | kipply's blog
kipp.ly · 5,670 words · saved by 17 readers
kipply's blog about stuff she does or reads about or observes
This article presents detailed few-principles reasoning about large language model inference performance, with no experiments or difficult math. The amount of understanding that can be acquired this way is really impressive and practical! A very simple model of latency for inference turns out to be a good fit for emprical results. It's helped me make better predictions and form better explanations about transformer inference. This post assumes some prior knowledge about transformers, say at having understood most of The Illustrated Transformer but not having internalised all of it. Familiarity
saved by
- Amir
- Pranav
- Varun Shenoy
- Claire Wang
- Daniel Bashir
- Ratan Kaliani
- HudZah
- Linda
- Luodi Wang
- Atem Aguer
- Tiffany Wang
- Omkaar
related reading
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- All About Transformer Inferencejax-ml.github.io
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Making Deep Learning go Brrrr From First Principleshorace.io
- All the Transformer Math You Need to Know | How To Scale Your Modeljax-ml.github.io
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- Transformer inference trickssubstack.com
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- How is LLaMa.cpp possible?finbarr.ca
- All About Rooflines | How To Scale Your Modeljax-ml.github.io
- Optimally allocating compute between inference and trainingepoch.ai
- Efficient LLM inferencefinbarrtimbers.substack.com