LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism
arxiv.org · 6,107 words · saved by 1 readers
N/A
LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism Diandian Gu Peng Sun Qinghao Hu School of Computer Science Sensetime Research & S-Lab, NTU & Peking University Shanghai AI Laboratory Shanghai AI…
related reading
- transformer_attention.pdfarxiv.org
- TokenRing: An Efficient Parallelism Framework for Infinite-Context LLMs via Bidirectional Communicationarxiv.org
- [2512.23675] End-to-End Test-Time Training for Long Contextarxiv.org
- Context Parallelism for Scalable Million-Token Inferencearxiv.org
- 2502.11089arxiv.org
- [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
- 2310.01889arxiv.org
- Star Attention: Efficient LLM Inference over Long Sequencesarxiv.org
- Log-Linear Attentionarxiv.org
- [2507.04239] Scaling Context Requires Rethinking Attentionarxiv.org
- Overleaf Examplearxiv.org
- 5D parallelism in LLM training - gdymind's Bloggdymind.com