[2106.08295] A White Paper on Neural Network Quantization

Abstract:While neural networks have advanced the frontiers in many applications, they often come at a high computational cost. Reducing the power and latency of neural network inference is key if we want to integrate modern networks into edge devices with strict power and compute requirements. Neural network quantization is one of the most effective ways of achieving these savings but the additional noise it induces can lead to accuracy degradation. In this white paper, we introduce state-of-the-art algorithms for mitigating the impact of quantization noise on the network's performance while maintaining low-bit weights and activations. We start with a hardware motivated introduction to quantization and then consider two main classes of algorithms: Post-Training Quantization (PTQ) and Quantization-Aware-Training (QAT). PTQ requires no re-training or labelled data and is thus a lightweight push-button approach to quantization. In most cases, PTQ is sufficient for achieving 8-bit quantization with close to floating-point accuracy. QAT requires fine-tuning and access to labeled training data but enables lower bit quantization with competitive results. For both solutions, we provide tested pipelines based on existing literature and extensive experimentation that lead to state-of-the-art performance for common deep learning models and tasks.

Abstract:While neural networks have advanced the frontiers in many applications, they often come at a high computational cost. Reducing the power and latency of neural network inference is key if we want to integrate modern networks into edge devices with strict power and compute requirements. Neural network quantization is one of the most effective ways of achieving these savings but the additional noise it induces can lead to accuracy degradation. In this white paper, we introduce state-of-the-art algorithms for mitigating the impact of quantization noise on the network's performance while m

related reading

• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationarxiv.org
• [1806.08342] Quantizing deep convolutional networks for efficient inference: A whitepaperarxiv.org
• Zoom In: An Introduction to Circuitsdistill.pub
• On neural scaling and the quanta hypothesisericjmichaud.com
• A Visual Guide to Quantization - by Maarten Grootendorstnewsletter.maartengrootendorst.com
• Achieving FP32 Accuracy for INT8 Inference Using Quantization Aware Training with NVIDIA TensorRT | NVIDIA Technical Blogdeveloper.nvidia.com
• Neural networks and deep learningneuralnetworksanddeeplearning.com
• A Guide to Quantization in LLMs | Symbl.aisymbl.ai
• SmoothQuant: Accurate and EfficientPost-Training Quantization for Large Language Modelsarxiv.org
• How To Optimize Computer Vision Models For Edge Devices | Picselliapicsellia.com
• [2101.01321] I-BERT: Integer-only BERT Quantizationarxiv.org
• Bayesian Neural Networkscs.toronto.edu