TokenRing: An Efficient Parallelism Framework for Infinite-Context LLMs via Bidirectional Communication
arxiv.org · 5,809 words · saved by 1 readers
N/A
TokenRing: An Efficient Parallelism Framework for Infinite-Context LLMs via Bidirectional Communication Zongwu Wang 1 2 Fangxin Liu* 1 2 Mingshuai Li 2 Li Jiang* 1 2 1. Department of Computer Science and Engineering, Shanghai Jiao Tong University, 2. Shanghai Qi Zhi Institute *Corresponding Author {wangzongwu, liufangxin, ljiang cs}@sjtu.edu.cn arXiv:2412.20501v1…
related reading
- Context Parallelism for Scalable Million-Token Inferencearxiv.org
- How To Scale Your Modeljax-ml.github.io
- LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelismarxiv.org
- 2310.01889arxiv.org
- Overleaf Examplearxiv.org
- Big Boss (@0xBADB01E) on Xx.com
- 5D parallelism in LLM training - gdymind's Bloggdymind.com
- Paradigms of Parallelism | Colossal-AIcolossalai.org
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
- Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com