TPUs vs GPUs for Transformers (BERT) — Tim Dettmers
Here I develop a theoretical model of TPUs vs GPUs for transformers as used by BERT and show that current GPUs are about 32% to 54% slower for this task.
On the computational side, there have been confusions about how TPUs and GPUs relate to BERT . BERT base was trained with 4 TPU pods (16 TPU chips) in 4 days and BERT large with 16 TPUs (64 TPU chips) in 4 days. Does this mean only Google can train a BERT model? Does this mean that GPUs are dead? There are two fundamental things to understand here: (1) A TPU is a matrix multiplication engine — it does matrix multiplication and matrix operations, but not much else. It is fast at computing matrix multiplication, but one has to understand that (2) the slowest thing in matrix multiplication is to
Explore this link on the map →related reading
- How To Scale Your Modeljax-ml.github.io
- How to Think About TPUs | How To Scale Your Modeljax-ml.github.io
- The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
- How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- All About Rooflines | How To Scale Your Modeljax-ml.github.io
- Tiny TPUtinytpu.com
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- TPU Deep Divehenryhmko.github.io
- The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
- Making Deep Learning go Brrrr From First Principleshorace.io
- The chip made for the AI inference era – the Google TPUuncoveralpha.com