open-infra-index/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md at main · deepseek-ai/open-infra-index
github.com · 927 words · saved by 1 readers
Production-tested AI infrastructure tools for efficient AGI development and community-driven innovation - deepseek-ai/open-infra-index
Day 6: One More Thing, DeepSeek-V3/R1 Inference System Overview System Design Principles The optimization objectives of serving DeepSeek-V3/R1 inference are: higher throughput and lower latency. To optimize these two objectives, our solution employs cross-node Expert Parallelism (EP). First, EP significantly scales the batch size, enhancing GPU matrix computation efficiency and boosting throughput. Second, EP distributes experts across GPUs, with each GPU processing only a small subset of experts (reducing memory access demands), thereby lowering latency. However, EP increases system complexit
related reading
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Best practices to accelerate inference for large-scale production workloadstogether.ai
- A Guide to AI Inference Engineering - ByteByteGo Newsletterblog.bytebytego.com
- Together AI | The AI Native Cloudtogether.ai
- DeepSeek-V3: A Large-Scale MoE Pretraining Benchmark for MLPerf Training v6.0mlcommons.org
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelsarxiv.org
- Real-time LLM Inference on Standard Datacenter GPUs (3,000 tokens/s per request)blog.kog.ai
- Kimi-K2.5 Inference Benchmark - Luminalluminal.com
- RL at 1T Scale: prime-rl Performance Deep Diveprimeintellect.ai
- How to Deploy Your Modelhtdym.sailresearch.com