HPC vs AI
System architecture There are a few differences between designing a supercomputer for AI and designing a supercomputer for traditional modeling and simulation.
System architecture There are a few differences between designing a supercomputer for AI and designing a supercomputer for traditional modeling and simulation. I once gave a presentation internally at Microsoft that contained this table: The biggest differences amount to: AI benefits from multi-plane fat trees. They make collectives faster since more nodes can talk to each other without hopping through switches, but they require more nodes and switches to be connected together in a small space. In practice, this requires using expensive optics instead of DAC cables. AI uses a lot of node-local
saved by
related reading
- How To Scale Your Modeljax-ml.github.io
- Multi-Datacenter Training: OpenAI's Ambitious Plan To Beat Google's Infrastructuresemianalysis.com
- AI Infrastructure Handbook – Nextraai-infra.guide
- AI Chip Architecturesjacobpeake.com
- Components of an Open Source AI Compute Tech Stackanyscale.com
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- My picture of the present in AI — LessWronglesswrong.com
- AI Chip Architecturesjepeake.com
- How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
- The Short Case for Nvidia Stock | YouTube Transcript Optimizeryoutubetranscriptoptimizer.com
- Network and Storage Benchmarks for LLM Training on the Cloud | Henry Zhumaknee.github.io
- A Hitchhiker’s Guide to ML Training Infrastructure | CMU Software Engineering Institutesei.cmu.edu