Memory bandwidth constraints imply economies of scale in AI inference — LessWrong

Contemporary GPUs often have very imbalanced memory vs arithmetic operation capabilities. For instance, an H100 can do around 3e15 8-bit FLOP/s, but…

x Memory bandwidth constraints imply economies of scale in AI inference — LessWrong Compute AI Frontpage 80 Memory bandwidth constraints imply economies of scale in AI inference by Ege Erdil 17th Sep 2023 4 min read 34 80 Contemporary GPUs often have very imbalanced memory vs arithmetic operation capabilities. For instance, an H100 can do around 3e15 8-bit FLOP/s, but the speed at which information can move between the cores and the GPU memory is only 3 TB/s. As 8 bits = 1 byte, there is a mismatch of three orders of magnitude between the arithmetic operation capabilities of the GPU and its me

related reading

• Transformer Inference Arithmetic | kipply's blogkipp.ly
• All About Rooflines | How To Scale Your Modeljax-ml.github.io
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
• Nvidia H100 GPUs: Supply and Demand · GPU Utils ⚡️gpus.llm-utils.org
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• How is LLaMa.cpp possible?finbarr.ca
• 2403.14123arxiv.org
• The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
• Strangely, Matrix Multiplications on GPUs Run Faster When Given "Predictable" Data! [short]thonking.ai
• How to Think About TPUs | How To Scale Your Modeljax-ml.github.io
• Can AI scaling continue through 2030? | Epoch AIepoch.ai