LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism
arxiv.org · 6,107 words · saved by 1 readers
N/A
LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism Diandian Gu Peng Sun Qinghao Hu School of Computer Science Sensetime Research & S-Lab, NTU & Peking University Shanghai AI Laboratory Shanghai AI…
related reading
- How To Scale Your Modeljax-ml.github.io
- DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelsarxiv.org
- TokenRing: An Efficient Parallelism Framework for Infinite-Context LLMs via Bidirectional Communicationarxiv.org
- 2310.01889arxiv.org
- Paradigms of Parallelism | Colossal-AIcolossalai.org
- 5D parallelism in LLM training - gdymind's Bloggdymind.com
- [2507.04239] Scaling Context Requires Rethinking Attentionarxiv.org
- Overleaf Examplearxiv.org
- 2305.19370arxiv.org
- Context Parallelism for Scalable Million-Token Inferencearxiv.org
- [2512.23675] End-to-End Test-Time Training for Long Contextarxiv.org
- Reimagining LLM Memory: Using Context as Training Data Unlocks Models That Learn at Test-Time | NVIDIA Technical Blogdeveloper.nvidia.com