Incredibly Fast BLOOM Inference with DeepSpeed and Accelerate

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

This article shows how to get an incredibly fast per token throughput when generating with the 176B parameter BLOOM model. As the model needs 352GB in bf16 (bfloat16) weights (176*2), the most efficient set-up is 8x80GB A100 GPUs. Also 2x8x40GB A100s or 2x8x48GB A6000 can be used. The main reason for using these GPUs is that at the time of this writing they provide the largest GPU memory, but other GPUs can be used as well. For example, 24x32GB V100s can be used. Using a single node will typically deliver a fastest throughput since most of the time intra-node GPU linking hardware is faster…

related reading

• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Best practices to accelerate inference for large-scale production workloadstogether.ai
• Real-time LLM Inference on Standard Datacenter GPUs (3,000 tokens/s per request)blog.kog.ai
• [2207.00032] DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalearxiv.org
• Together AI | The AI Native Cloudtogether.ai
• How to Deploy Your Modelhtdym.sailresearch.com
• Accelerated Inference for Large Transformer Models Using NVIDIA Triton Inference Server | NVIDIA Technical Blogdeveloper.nvidia.com
• Optimizing inference · Hugging Facehuggingface.co
• Two different tricks for fast LLM inferenceseangoedecke.com
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• Fast Inference from Transformers via Speculative Decodingarxiv.org