✳flâneur — a map of the web's best reading
Memory bandwidth constraints imply economies of scale in AI inference — LessWrong
lesswrong.com · 5,211 words · saved by 1 readers
Contemporary GPUs often have very imbalanced memory vs arithmetic operation capabilities. For instance, an H100 can do around 3e15 8-bit FLOP/s, but…
x Memory bandwidth constraints imply economies of scale in AI inference — LessWrong Compute AI Frontpage 80 Memory bandwidth constraints imply economies of scale in AI inference by Ege Erdil 17th Sep 2023 4 min read 34 80 Contemporary GPUs often have very imbalanced memory vs arithmetic operation capabilities. For instance, an H100 can do around 3e15 8-bit FLOP/s, but the speed at which information can move between the cores and the GPU memory is only 3 TB/s. As 8 bits = 1 byte, there is a mismatch of three orders of magnitude between the arithmetic operation capabilities of the GPU and its me
Explore this link on the map →related reading
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- The Inference Shift – Stratechery by Ben Thompsonstratechery.com
- Making Deep Learning go Brrrr From First Principleshorace.io
- The Short Case for Nvidia Stock | YouTube Transcript Optimizeryoutubetranscriptoptimizer.com
- My picture of the present in AI — LessWronglesswrong.com
- How is LLaMa.cpp possible?finbarr.ca
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Real-time LLM Inference on Standard Datacenter GPUs (3,000 tokens/s per request)blog.kog.ai
- A guide to LLM inference and performancebaseten.co
- Why AGI Will Not Happen - Tim Dettmerstimdettmers.com
- Navigating the High Cost of AI Compute | Andreessen Horowitza16z.com