All About Transformer Inference | How To Scale Your Model
Performing inference on a Transformer can be very different from training. Partly this is because inference adds a new factor to consider: latency. In this section, we will go all the way from sampling a single new token from a model to efficiently scaling a large Transformer across many slices of accelerators as part of an inference engine.
The Basics of Transformer Inference So you’ve trained a Transformer, and you want to use it to generate some new sequences. At the end of the day, benchmark scores going up and loss curves going down are only proxies for whether something interesting is going to happen once the rubber hits the road!Historically, you can do a surprising amount of research on Transformers without ever touching inference — scoring-based multiple choice benchmarks can be run efficiently without a proper KV cache or generation loop implementation. This meant, especially in research codebases, there's often a lot…
saved by
related reading
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- How To Scale Your Modeljax-ml.github.io
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- Efficiently Scaling Transformer Inferencearxiv.org
- Transformer inference trickssubstack.com
- All the Transformer Math You Need to Know | How To Scale Your Modeljax-ml.github.io
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Fast Inference from Transformers via Speculative Decodingarxiv.org