✳flâneur — a map of the web's best reading
Transformer Inference Arithmetic | kipply's blog
kipp.ly · 5,670 words · saved by 17 readers
kipply's blog about stuff she does or reads about or observes
This article presents detailed few-principles reasoning about large language model inference performance, with no experiments or difficult math. The amount of understanding that can be acquired this way is really impressive and practical! A very simple model of latency for inference turns out to be a good fit for emprical results. It's helped me make better predictions and form better explanations about transformer inference. This post assumes some prior knowledge about transformers, say at having understood most of The Illustrated Transformer but not having internalised all of it. Familiarity
Explore this link on the map →saved by
- Amir
- Pranav
- Varun Shenoy
- Claire Wang
- Daniel Bashir
- Ratan Kaliani
- HudZah
- Linda
- Luodi Wang
- Atem Aguer
- Tiffany Wang
- Omkaar
related reading
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Making Deep Learning go Brrrr From First Principleshorace.io
- All the Transformer Math You Need to Know | How To Scale Your Modeljax-ml.github.io
- How is LLaMa.cpp possible?finbarr.ca
- How To Scale Your Modeljax-ml.github.io
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- Overleaf Examplearxiv.org
- All About Rooflines | How To Scale Your Modeljax-ml.github.io
- A guide to LLM inference and performancebaseten.co