Quantization

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Quantization is a technique to reduce the computational and memory costs of running inference by representing the weights and activations with low-precision data types like 8-bit integer (int8) instead of the usual 32-bit floating point (float32). Reducing the number of bits means the resulting model requires less memory storage, consumes less energy (in theory), and operations like matrix multiplication can be performed much faster with integer arithmetic. It also allows to run models on embedded devices, which sometimes only support integer data types. Theory The basic idea behind…

related reading

• A Visual Guide to Quantization - by Maarten Grootendorstnewsletter.maartengrootendorst.com
• lec05.pdfdropbox.com
• Quantization from the ground upngrok.com
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationarxiv.org
• A Guide to Quantization in LLMs | Symbl.aisymbl.ai
• The 4-bitter Lesson | humans&humansand.ai
• Achieving FP32 Accuracy for INT8 Inference Using Quantization Aware Training with NVIDIA TensorRT | NVIDIA Technical Blogdeveloper.nvidia.com
• Efficient LLM inferencefinbarrtimbers.substack.com
• Do Qwen3.6 27B quantizations break the pelican?quesma.com
• lec06.pdfdropbox.com
• [2106.08295] A White Paper on Neural Network Quantizationarxiv.org
• 2112.10769arxiv.org