Memory bandwidth constraints imply economies of scale in AI inference — LessWrong
lesswrong.com · 5,211 words · saved by 1 readers
Contemporary GPUs often have very imbalanced memory vs arithmetic operation capabilities. For instance, an H100 can do around 3e15 8-bit FLOP/s, but…
x Memory bandwidth constraints imply economies of scale in AI inference — LessWrong Compute AI Frontpage 80 Memory bandwidth constraints imply economies of scale in AI inference by Ege Erdil 17th Sep 2023 4 min read 34 80 Contemporary GPUs often have very imbalanced memory vs arithmetic operation capabilities. For instance, an H100 can do around 3e15 8-bit FLOP/s, but the speed at which information can move between the cores and the GPU memory is only 3 TB/s. As 8 bits = 1 byte, there is a mismatch of three orders of magnitude between the arithmetic operation capabilities of the GPU and its me
related reading
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- The Inference Shift – Stratechery by Ben Thompsonstratechery.com
- Making Deep Learning go Brrrr From First Principleshorace.io
- The Short Case for Nvidia Stock | YouTube Transcript Optimizeryoutubetranscriptoptimizer.com
- 2403.14123arxiv.org
- My picture of the present in AI — LessWronglesswrong.com
- How is LLaMa.cpp possible?finbarrtimbers.substack.com
- How is LLaMa.cpp possible?finbarr.ca
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- AI's Hardware Problemasianometry.substack.com
- Dylan Patel — Deep dive on the 3 big bottlenecks to scaling AI computedwarkesh.com