HPC vs AI
System architecture There are a few differences between designing a supercomputer for AI and designing a supercomputer for traditional modeling and simulation.
System architecture There are a few differences between designing a supercomputer for AI and designing a supercomputer for traditional modeling and simulation. I once gave a presentation internally at Microsoft that contained this table: The biggest differences amount to: AI benefits from multi-plane fat trees. They make collectives faster since more nodes can talk to each other without hopping through switches, but they require more nodes and switches to be connected together in a small space. In practice, this requires using expensive optics instead of DAC cables. AI uses a lot of node-local
saved by
related reading
- How To Scale Your Modeljax-ml.github.io
- AI Chip Architecturesjacobpeake.com
- Navigating the High Cost of AI Compute | Andreessen Horowitza16z.com
- Multi-Datacenter Training: OpenAI's Ambitious Plan To Beat Google's Infrastructuresemianalysis.com
- How to Build the Future of AI in the United States | IFPifp.org
- Computational Power and AI - AI Now Instituteainowinstitute.org
- AI Chip Architecturesjepeake.com
- Network and Storage Benchmarks for LLM Training on the Cloud | Henry Zhumaknee.github.io
- The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
- How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
- Navigating the High Cost of AI Compute | Andreessen Horowitza16z.com
- A basic systems architecture for AI agents that do autonomous research — LessWronglesswrong.com