Redesigning the Inference Chip: From Nvidia GPU's Flaws to OpenAI Jalapeño
zartbot.github.io · 9,714 words · saved by 1 readers
First principles · Core Slice architecture · Memory subsystem · On-chip network · Software stack & design loop
TL;DR If I were reborn as the Infra lead of some foundation-model team, I would certainly end up with all kinds of complaints about Nvidia's GPUs and the system as a whole, and then go build my own chip... For Nvidia's networking part there is nothing to discuss: RoCE has all sorts of defects, and the DPU also comes with a pile of performance and security problems. No wonder Nvidia has started hyping Scale-in again this year, but for a team that has never run a cloud, it will take at least 5 more years to get mature in this area. As an example, we recently did something in our MaaS online…
saved by
related reading
- Ex-NVIDIA Engineer: Why AI Is About to Get 1000x Cheaperyoutube.com
- Meta harness makes 10 times better Kimi K3 chipluoluo.ai
- OpenAI Jalapeño: Better Than Nvidia Blackwellnewsletter.semianalysis.com
- AI Chip Architecturesjacobpeake.com
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- AI Chip Architecturesjepeake.com
- How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
- Interview with Richard Ho, OpenAImorethanmoore.substack.com
- Chip design from the bottom up – Reiner Popeyoutube.com
- The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
- TPU Deep Divehenryhmko.github.io
- Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com