Incredibly Fast BLOOM Inference with DeepSpeed and Accelerate

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

This article shows how to get an incredibly fast per token throughput when generating with the 176B parameter BLOOM model. As the model needs 352GB in bf16 (bfloat16) weights (176*2), the most efficient set-up is 8x80GB A100 GPUs. Also 2x8x40GB A100s or 2x8x48GB A6000 can be used. The main reason for using these GPUs is that at the time of this writing they provide the largest GPU memory, but other GPUs can be used as well. For example, 24x32GB V100s can be used. Using a single node will typically deliver a fastest throughput since most of the time intra-node GPU linking hardware is faster…

related reading

• Transformer Inference Arithmetic | kipply's blogkipp.ly
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• All About Rooflines | How To Scale Your Modeljax-ml.github.io
• The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• All About Transformer Inferencejax-ml.github.io
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• Best practices to accelerate inference for large-scale production workloadstogether.ai
• How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
• Pipeline-Parallelism: Distributed Training via Model Partitioningsiboehm.com
• [2207.00032] DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalearxiv.org