Achieving FP32 Accuracy for INT8 Inference Using Quantization Aware Training with NVIDIA TensorRT | NVIDIA Technical Blog

Deep learning is revolutionizing the way that industries are delivering products and services. These services include object detection, classification, and segmentation for computer vision, and text extraction, classification, and summarization for language-based applications. These applications must run in real time. Most of the models are trained in floating-point 32-bit arithmetic to take advantage of a wider dynamic range. However, at inference, these models may take a longer time to predict results compared to reduced precision inference, causing some delay in the real-time responses, and affecting the user experience. It’s better in many cases to use reduced precision or 8-bit integer numbers. The challenge is that simply rounding the weights after training may result in a lower accuracy model, especially if the weights have a wide dynamic range. This post provides a simple introduction to quantization-aware training (QAT), and how to implement fake-quantization during training,

Achieving FP32 Accuracy for INT8 Inference Using Quantization Aware Training with NVIDIA TensorRT | NVIDIA Technical Blog Technical Blog Subscribe Related Resources Computer Vision / Video Analytics English 中文 Achieving FP32 Accuracy for INT8 Inference Using Quantization Aware Training with NVIDIA TensorRT Jul 20, 2021 By Neta Zmora , Hao Wu and Jay Rodge Like Discuss (1) L T F R E AI-Generated Summary Like Dislike Quantization-aware training (QAT) is a technique that includes quantization error in the training phase to improve the accuracy of quantized models, enabling the network to adapt to

related reading

• A Visual Guide to Quantization - by Maarten Grootendorstnewsletter.maartengrootendorst.com
• What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Studyarxiv.org
• [1806.08342] Quantizing deep convolutional networks for efficient inference: A whitepaperarxiv.org
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationarxiv.org
• A Guide to Quantization in LLMs | Symbl.aisymbl.ai
• [2101.01321] I-BERT: Integer-only BERT Quantizationarxiv.org
• The 4-bitter Lesson | humans&humansand.ai
• Composer2.pdfcursor.com
• Quantization · Hugging Facehuggingface.co
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• SmoothQuant: Accurate and EfficientPost-Training Quantization for Large Language Modelsarxiv.org
• [2106.08295] A White Paper on Neural Network Quantizationarxiv.org