[1806.08342] Quantizing deep convolutional networks for efficient inference: A whitepaper

We present an overview of techniques for quantizing convolutional neural networks for inference with integer weights and activations. Per-channel quantization of weights and per-layer quantization of activations to 8-bits of precision post-training produces classification accuracies within 2% of floating point networks for a wide variety of CNN architectures. Model sizes can be reduced by a factor of 4 by quantizing weights to 8-bits, even when 8-bit arithmetic is not supported. This can be achieved with simple, post training quantization of weights.We benchmark latencies of quantized networks on CPUs and DSPs and observe a speedup of 2x-3x for quantized implementations compared to floating point on CPUs. Speedups of up to 10x are observed on specialized processors with fixed point SIMD capabilities, like the Qualcomm QDSPs with HVX. Quantization-aware training can provide further improvements, reducing the gap to floating point to 1% at 8-bit precision. Quantization-aware training also allows for reducing the precision of weights to four bits with accuracy losses ranging from 2% to 10%, with higher accuracy drop for smaller networks.We introduce tools in TensorFlow and TensorFlowLite for quantizing convolutional networks and review best practices for quantization-aware training to obtain high accuracy with quantized weights and activations. We recommend that per-channel quantization of weights and per-layer quantization of activations be the preferred quantization scheme for hardware acceleration and kernel optimization. We also propose that future processors and hardware accelerators for optimized inference support precisions of 4, 8 and 16 bits.

[1806.08342] Quantizing deep convolutional networks for efficient inference: A whitepaper Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Machine Learning arXiv:1806.08342 (cs) [Submitted on 21 Jun 2018] Title: Quantizing deep convolutional networks for efficient inference: A whitepaper Authors: Raghuraman Krishnamoorthi View a PDF of the paper titled Quantizing deep convolutional networks for efficient inference: A whitepaper, by Raghuraman Krishnamoorthi View PDF Abstract: We pre

related reading

• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationarxiv.org
• [2106.08295] A White Paper on Neural Network Quantizationarxiv.org
• Achieving FP32 Accuracy for INT8 Inference Using Quantization Aware Training with NVIDIA TensorRT | NVIDIA Technical Blogdeveloper.nvidia.com
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• A Visual Guide to Quantization - by Maarten Grootendorstnewsletter.maartengrootendorst.com
• A Guide to Quantization in LLMs | Symbl.aisymbl.ai
• CS231n Deep Learning for Computer Visioncs231n.github.io
• [1905.11946] EfficientNet: Rethinking Model Scaling for Convolutional Neural Networksarxiv.org
• Convolutional Neural Networks, Explained | Towards Data Sciencetowardsdatascience.com
• SmoothQuant: Accurate and EfficientPost-Training Quantization for Large Language Modelsarxiv.org
• Quantization · Hugging Facehuggingface.co
• The Little Book of Deep Learningfleuret.org