Estimating Training Compute of Deep Learning Models – Epoch

We describe two approaches for estimating the training compute of Deep Learning systems, by counting operations and looking at GPU time. ML Models trained on more compute have better performance and more advanced capabilities (see e.g. Kaplan et al., 2020 or Hoffman et al., 2022). Due to this, estimating and reporting compute usage is crucial to enable accurate comparisons between ML models. Compute usage is commonly measured as the number of floating point operations (FLOP) required to train the final version of the system. To estimate this we can resort to two strategies: a) using information about the architecture and amount of training data, or b) using information about the hardware used and training time. Below we provide two calculators that illustrate these methods. Copy and paste the following information in your paper Copy and paste the following information in your paper Do you see a mistake or do you want to submit missing information about hardware specs? Fill this form an

Estimating training compute of deep learning models | Epoch AI ML Models trained on more compute have better performance and more advanced capabilities (see e.g. Kaplan et al., 2020 or Hoffman et al., 2022 ). Due to this, estimating and reporting compute usage is crucial to enable accurate comparisons between ML models. Compute usage is commonly measured as the number of floating point operations (FLOP) required to train the final version of the system. To estimate this we can resort to two strategies: a) using information about the architecture and amount of training data, or b) using informa

saved by

• Pier-Luc Nadeau-Labadie
• Danish

related reading

• Making Deep Learning go Brrrr From First Principleshorace.io
• Latest | Epoch AIepochai.org
• The Little Book of Deep Learningfleuret.org
• Fermi estimate of future training runsdanieldewey.net
• How To Scale Your Modeljax-ml.github.io
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• All About Rooflines | How To Scale Your Modeljax-ml.github.io
• Optimally allocating compute between inference and training | Epoch AIepochai.org
• GPU Performance Background User's Guide - NVIDIA Docsdocs.nvidia.com
• Compute trends across three eras of machine learning | Epoch AIepochai.org
• The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com