open-infra-index/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md at main · deepseek-ai/open-infra-index

Production-tested AI infrastructure tools for efficient AGI development and community-driven innovation - deepseek-ai/open-infra-index

Day 6: One More Thing, DeepSeek-V3/R1 Inference System Overview System Design Principles The optimization objectives of serving DeepSeek-V3/R1 inference are: higher throughput and lower latency. To optimize these two objectives, our solution employs cross-node Expert Parallelism (EP). First, EP significantly scales the batch size, enhancing GPU matrix computation efficiency and boosting throughput. Second, EP distributes experts across GPUs, with each GPU processing only a small subset of experts (reducing memory access demands), thereby lowering latency. However, EP increases system complexit

related reading

• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
• Best practices to accelerate inference for large-scale production workloadstogether.ai
• A Guide to AI Inference Engineering - ByteByteGo Newsletterblog.bytebytego.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Composer2.pdfcursor.com
• Real-time LLM Inference on Standard Datacenter GPUs (3,000 tokens/s per request)blog.kog.ai
• Kimi-K2.5 Inference Benchmark - Luminalluminal.com
• The Short Case for Nvidia Stock | YouTube Transcript Optimizeryoutubetranscriptoptimizer.com
• [2207.00032] DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalearxiv.org
• Speculative Decoding - philkravphilkrav.com
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io