A Guide to Quantization in LLMs | Symbl.ai - https://symbl.ai/developers/blog/a-guide-to-quantization-in-llms/

The capabilities of Large Language Models (LLMs) have grown in leaps and bounds in recent years, making them more user-friendly and applicable in a growing number of use cases. However, as LLMs have increased in intelligence and complexity, the number of parameters, or weights and activations, i.e., its capacity to learn from and process data, has also grown. For example, GPT-3.5 has around 175 billion parameters, while the current state-of-the-art GPT-4 has in excess of 1 trillion parameters. However, the larger an LLM, the more memory it requires. This means that it is only feasible to run LLMs on high-specification hardware with the prerequisite amount of GPUs – this limits deployment options and, consequently, how readily LLM-based solutions can be adopted. Fortunately, machine learning researchers are devising a growing range of solutions to meet the challenge of growing model sizes – with one of the most prominent being quantization. In this guide, we explore the concept of quant

A Guide to Quantization in LLMs by Kartik Talamadupula | 02-21-2024 The capabilities of Large Language Models (LLMs) have grown in leaps and bounds in recent years, making them more user-friendly and applicable in a growing number of use cases. However, as LLMs have increased in intelligence and complexity, the number of parameters, or weights and activations, i.e., its capacity to learn from and process data, has also grown. For example, GPT-3.5 has around 175 billion parameters, while the current state-of-the-art GPT-4 has in excess of 1 trillion parameters. However, the larger an LLM, the m

saved by

• Eric Yoon

related reading

• Quantization from the ground upngrok.com
• Efficient LLM inferencefinbarrtimbers.substack.com
• A Visual Guide to Quantization - by Maarten Grootendorstnewsletter.maartengrootendorst.com
• 2305.14314arxiv.org
• AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Accelerationarxiv.org
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Studyarxiv.org
• Quantization and Hardware Architecture Co-Design for Matrix-Vector Multiplications of Large Language Modelsieeexplore.ieee.org
• SmoothQuant: Accurate and EfficientPost-Training Quantization for Large Language Modelsarxiv.org
• [2510.11696] QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMsarxiv.org
• 2210.17323arxiv.org
• The 4-bitter Lesson | humans&humansand.ai