A Visual Guide to Quantization - by Maarten Grootendorst

Exploring memory-efficient techniques for LLMs

A Visual Guide to Quantization Demystifying the Compression of Large Language Models Maarten Grootendorst Jul 22, 2024 524 26 44 Share Translations - Korean - Chinese - French As their name suggests, Large Language Models (LLMs) are often too large to run on consumer hardware. These models may exceed billions of parameters and generally need GPUs with large amounts of VRAM to speed up inference. As such, more and more research has been focused on making these models smaller through improved training, adapters, etc. One major technique in this field is called quantization . In this post, I will

related reading

• A Guide to Quantization in LLMs | Symbl.aisymbl.ai
• Quantization from the ground upngrok.com
• Quantization · Hugging Facehuggingface.co
• Efficient LLM inferencefinbarrtimbers.substack.com
• The 4-bitter Lesson | humans&humansand.ai
• lec06.pdfdropbox.com
• What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Studyarxiv.org
• 2112.10769arxiv.org
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationarxiv.org
• SmoothQuant: Accurate and EfficientPost-Training Quantization for Large Language Modelsarxiv.org
• AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Accelerationarxiv.org
• 2305.14314arxiv.org