Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation

Quantization techniques can reduce the size of Deep Neural Networks and improve inference latency and throughput by taking advantage of high throughput integer instructions. In this paper we review the mathematical aspects of quantization parameters and evaluate their choices on a wide range of neural network models for different application domains, including vision, speech, and language. We focus on quantization techniques that are amenable to acceleration by processors with high-throughput integer math pipelines. We also present a workflow for 8-bit quantization that is able to maintain accuracy within 1% of the floating-point baseline on all networks studied, including models that are more difficult to quantize, such as MobileNets and BERT-large.

Quantization techniques can reduce the size of Deep Neural Networks and improve inference latency and throughput by taking advantage of high throughput integer instructions. In this paper we review the mathematical aspects of quantization parameters and evaluate their choices on a wide range of neural network models for different application domains, including vision, speech, and language. We focus on quantization techniques that are amenable to acceleration by processors with high-throughput integer math pipelines. We also present a workflow for 8-bit quantization that is able to maintain acc

related reading

• [1806.08342] Quantizing deep convolutional networks for efficient inference: A whitepaperarxiv.org
• [2106.08295] A White Paper on Neural Network Quantizationarxiv.org
• A Visual Guide to Quantization - by Maarten Grootendorstnewsletter.maartengrootendorst.com
• On neural scaling and the quanta hypothesisericjmichaud.com
• A Guide to Quantization in LLMs | Symbl.aisymbl.ai
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Achieving FP32 Accuracy for INT8 Inference Using Quantization Aware Training with NVIDIA TensorRT | NVIDIA Technical Blogdeveloper.nvidia.com
• The Little Book of Deep Learningfleuret.org
• What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Studyarxiv.org
• [2101.01321] I-BERT: Integer-only BERT Quantizationarxiv.org
• Quantization · Hugging Facehuggingface.co
• How is LLaMa.cpp possible?finbarr.ca