DeepSeek-V3: A Large-Scale MoE Pretraining Benchmark for MLPerf Training v6.0 - MLCommons

MLPerf Training v6.0 introduces a large-scale pretraining benchmark built on DeepSeek-V3, bringing Mixture-of-Experts (MoE) evaluation to the suite.

Motivation and Architectural Relevance As Large Language Model (LLM) development increasingly adopts sparse computation, the benchmarks used to evaluate training performance need to keep pace. MLPerf™ Training v6.0 adds a large-scale pretraining benchmark built on DeepSeek-V3, a Mixture-of-Experts (MoE) architecture with 671B total parameters, of which 37B are activated per token. This benchmark captures the performance of critical innovations now standard in the industry, including Multi-head Latent Attention (MLA) and auxiliary-loss–free load balancing. Technical Architecture &…

saved by

• Arya G

related reading

• [2607.16051] Loop the Loopies!arxiv.org
• [2509.14786] Pre-training under infinite computearxiv.org
• The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
• DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelsarxiv.org
• DeepSeek-R1arxiv.org
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• >10x More Efficient Pretraining — Magicmagic.dev
• The Complete Guide to DeepSeek Models: V3, R1, V4 and Beyondbentoml.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• How Attention Got So Efficient [GQA/MLA/DSA]youtube.com
• DeepSeek: The View from Chinachinatalk.media
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org