A Visual Guide to Quantization - by Maarten Grootendorst

Exploring memory-efficient techniques for LLMs

A Visual Guide to Quantization Demystifying the Compression of Large Language Models Maarten Grootendorst Jul 22, 2024 524 26 44 Share Translations - Korean - Chinese - French As their name suggests, Large Language Models (LLMs) are often too large to run on consumer hardware. These models may exceed billions of parameters and generally need GPUs with large amounts of VRAM to speed up inference. As such, more and more research has been focused on making these models smaller through improved training, adapters, etc. One major technique in this field is called quantization . In this post, I will

related reading

• Quantization from the ground upngrok.com
• A Guide to Quantization in LLMs | Symbl.aisymbl.ai
• Efficient LLM inferencefinbarrtimbers.substack.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Compression and Intelligencegreene.sh
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• Deep Dive into LLMs like ChatGPTyoutube.com
• SmoothQuant: Accurate and EfficientPost-Training Quantization for Large Language Modelsarxiv.org
• A History of Large Language Modelsgregorygundersen.com
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformersarxiv.org