Transformer Inference Arithmetic | kipply's blog
kipp.ly · 5,670 words · saved by 17 readers
kipply's blog about stuff she does or reads about or observes
This article presents detailed few-principles reasoning about large language model inference performance, with no experiments or difficult math. The amount of understanding that can be acquired this way is really impressive and practical! A very simple model of latency for inference turns out to be a good fit for emprical results. It's helped me make better predictions and form better explanations about transformer inference. This post assumes some prior knowledge about transformers, say at having understood most of The Illustrated Transformer but not having internalised all of it. Familiarity
saved by
- Amir
- Pranav
- Varun Shenoy
- Claire Wang
- Daniel Bashir
- Ratan Kaliani
- HudZah
- Linda
- Luodi Wang
- Atem Aguer
- Tiffany Wang
- Omkaar
related reading
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- Making Deep Learning go Brrrr From First Principleshorace.io
- How To Scale Your Modeljax-ml.github.io
- All the Transformer Math You Need to Know | How To Scale Your Modeljax-ml.github.io
- How is LLaMa.cpp possible?finbarr.ca
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- All About Transformer Inferencejax-ml.github.io
- Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- Overleaf Examplearxiv.org
- All About Rooflines | How To Scale Your Modeljax-ml.github.io