open-infra-index/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md at main · deepseek-ai/open-infra-index

Production-tested AI infrastructure tools for efficient AGI development and community-driven innovation - deepseek-ai/open-infra-index

Day 6: One More Thing, DeepSeek-V3/R1 Inference System Overview System Design Principles The optimization objectives of serving DeepSeek-V3/R1 inference are: higher throughput and lower latency. To optimize these two objectives, our solution employs cross-node Expert Parallelism (EP). First, EP significantly scales the batch size, enhancing GPU matrix computation efficiency and boosting throughput. Second, EP distributes experts across GPUs, with each GPU processing only a small subset of experts (reducing memory access demands), thereby lowering latency. However, EP increases system complexit

related reading

• DeepSeek-R1arxiv.org
• Building a web search engine from scratch in two months with 3 billion neural embeddingsblog.wilsonl.in
• DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelsarxiv.org
• DeepSeek-V3: A Large-Scale MoE Pretraining Benchmark for MLPerf Training v6.0mlcommons.org
• DeepSeek: The View from Chinachinatalk.media
• Best practices to accelerate inference for large-scale production workloadstogether.ai
• The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• DeepSeek R1's recipe to replicate o1 and the future of reasoning LMsinterconnects.ai
• Deepseek: The Quiet Giant Leading China’s AI Racechinatalk.media
• Open-R1: a fully open reproduction of DeepSeek-R1huggingface.co
• The Complete Guide to DeepSeek Models: V3, R1, V4 and Beyondbentoml.com