Not All Open Model Tokens Are the Same | YC Paper Club - YouTube
This week’s Paper Club is all about inference. For every FLOP spent training a model, roughly six go to serving it each year — and getting that part right ha...
This week’s Paper Club is all about inference. For every FLOP spent training a model, roughly six go to serving it each year — and getting that part right has produced over $220 billion in market cap in the last decade. But inference is much more than calling model.forward. Every token has to move through a complicated stack of routing, caching, scheduling, kernels, memory, and hardware, all while balancing speed, cost, and quality. We brought together researchers and builders from Together AI, Ollama, Luminal, and SGLang & RadixArk to take us through that stack from end to end. We’ll see…
saved by
related reading
- Stanford CS25: Transformers United V6 I Serving Transformers: Lessons from the Trenchesyoutube.com
- What's New in Inference Engineering — Philip Kiely, Basetenyoutube.com
- How trillion-parameter AI models work and how to serve themyoutube.com
- How GPT, Claude, and Gemini are actually trained and served – Reiner Popeyoutube.com
- Why Sam Altman is against a single superintelligenceyoutube.com
- Recursive Language Models — Alex Zhang, MIT PhDyoutube.com
- Inference, Diffusion, World Models, and More | YC Paper Clubyoutube.com
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- Spending Inference Time - Kevin Lukevinlu.ai
- How To Scale Your Modeljax-ml.github.io
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- All About Transformer Inferencejax-ml.github.io