A Guide to Quantization in LLMs | Symbl.ai - https://symbl.ai/developers/blog/a-guide-to-quantization-in-llms/

The capabilities of Large Language Models (LLMs) have grown in leaps and bounds in recent years, making them more user-friendly and applicable in a growing number of use cases. However, as LLMs have increased in intelligence and complexity, the number of parameters, or weights and activations, i.e., its capacity to learn from and process data, has also grown. For example, GPT-3.5 has around 175 billion parameters, while the current state-of-the-art GPT-4 has in excess of 1 trillion parameters. However, the larger an LLM, the more memory it requires. This means that it is only feasible to run LLMs on high-specification hardware with the prerequisite amount of GPUs – this limits deployment options and, consequently, how readily LLM-based solutions can be adopted. Fortunately, machine learning researchers are devising a growing range of solutions to meet the challenge of growing model sizes – with one of the most prominent being quantization. In this guide, we explore the concept of quant

A Guide to Quantization in LLMs by Kartik Talamadupula | 02-21-2024 The capabilities of Large Language Models (LLMs) have grown in leaps and bounds in recent years, making them more user-friendly and applicable in a growing number of use cases. However, as LLMs have increased in intelligence and complexity, the number of parameters, or weights and activations, i.e., its capacity to learn from and process data, has also grown. For example, GPT-3.5 has around 175 billion parameters, while the current state-of-the-art GPT-4 has in excess of 1 trillion parameters. However, the larger an LLM, the m

saved by

• Eric Yoon

related reading

• A Visual Guide to Quantization - by Maarten Grootendorstnewsletter.maartengrootendorst.com
• SmoothQuant: Accurate and EfficientPost-Training Quantization for Large Language Modelsarxiv.org
• In-depth guide to fine-tuning LLMs with LoRA and QLoRA | Mercity Researchmercity.ai
• What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Studyarxiv.org
• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• Quantization · Hugging Facehuggingface.co
• LLM.int8()arxiv.org
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• [2306.07629] SqueezeLLM: Dense-and-Sparse Quantizationarxiv.org
• Optimizing inference · Hugging Facehuggingface.co
• The Llama Hitchiking Guide to Local LLMs – hackerllamaosanseviero.github.io
• Finetune LLMs on your own consumer hardware using tools from PyTorch and Hugging Face ecosystem – PyTorchpytorch.org