Context Parallelism for Scalable Million-Token Inference

N/A

C ONTEXT PARALLELISM FOR S CALABLE M ILLION -T OKEN I NFERENCE Amy (Jie) Yang 1 Jingyi Yang 1 Aya Ibrahim 1 Xinfeng Xie 1 Bangsheng Tang 1 Grigory Sizov 1 Jeremy Reizenstein 1 Jongsoo Park 1 Jianyu Huang 1 A BSTRACT We present context parallelism for long-context large language model inference, which achieves near-linear…

related reading

• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Parallel CPU-GPU Execution for LLM Inference on Constrained GPUsarxiv.org
• TokenRing: An Efficient Parallelism Framework for Infinite-Context LLMs via Bidirectional Communicationarxiv.org
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• [2412.02252] Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarityarxiv.org
• [2512.23675] End-to-End Test-Time Training for Long Contextarxiv.org
• Overleaf Examplearxiv.org
• All About Transformer Inferencejax-ml.github.io
• 2502.11089arxiv.org
• [2506.06266] Cartridges: Lightweight and general-purpose long context representations via self-studyarxiv.org