Reducing Activation Recomputation
arxiv.org · 8,193 words · saved by 1 readers
N/A
Reducing Activation Recomputation in Large Transformer Models Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro NVIDIA arXiv:2205.05198v1 [cs.LG] 10 May 2022 Abstract…
related reading
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- transformer_attention.pdfarxiv.org
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- Linear Transformers Are Faster After All – Manifest AImanifestai.com
- The Annotated Transformernlp.seas.harvard.edu
- [2205.14135] FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenessarxiv.org
- 2305.19370arxiv.org
- [1909.08053] Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismarxiv.org
- 5D parallelism in LLM training - gdymind's Bloggdymind.com
- Fast Inference from Transformers via Speculative Decodingarxiv.org