What Limits Performance of (PyTorch) Neural Networks when running on a CPU? | B. Nikolic Software and Computing Blog

This is a little experiment to use CPU performance monitoring counters to find out what limits the maximum performance of PyTorch Neural Networks when running on a CPU.

This is a little experiment to use CPU performance monitoring counters to find out what limits the maximum performance of PyTorch Neural Networks when running on a CPU. The experiment was carried out in the following way: Performance metrics were calculated as described in the post describing counting PMU events in Python and the post applying this to PyTorch . Measurements were made for all torchvision models, and repeated 10 times to obtain an average. The variety of models is used in order to have a range of algorithms so that the limiting factor becomes more apparent. Measurements were mad

related reading

• Making Deep Learning go Brrrr From First Principleshorace.io
• Counting FLOPS in PyTorch using CPU PMU counters | B. Nikolic Software and Computing Blogbnikolic.co.uk
• Counting FLOPS in PyTorch using CPU PMU counters | B. Nikolic Software and Computing Blogbnikolic.co.uk
• All About Rooflines | How To Scale Your Modeljax-ml.github.io
• Strangely, Matrix Multiplications on GPUs Run Faster When Given "Predictable" Data! [short]thonking.ai
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklogsiboehm.com
• The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
• Could a Neuroscientist Understand a Microprocessor? | PLOS Computational Biologyjournals.plos.org
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• PiTorch: ML on Baremetal Raspberry Pis | projectsmasonjwang.com
• Estimating training compute of deep learning models | Epoch AIepochai.org