flâneur

A Guide to Quantization in LLMs | Symbl.ai - https://symbl.ai/developers/blog/a-guide-to-quantization-in-llms/

symbl.ai · 2,545 words · saved by 1 readers

The capabilities of Large Language Models (LLMs) have grown in leaps and bounds in recent years, making them more user-friendly and applicable in a growing number of use cases. However, as LLMs have increased in intelligence and complexity, the number of parameters, or weights and activations, i.e., its capacity to learn from and process data, has also grown. For example, GPT-3.5 has around 175 billion parameters, while the current state-of-the-art GPT-4 has in excess of 1 trillion parameters. However, the larger an LLM, the more memory it requires. This means that it is only feasible to run LLMs on high-specification hardware with the prerequisite amount of GPUs – this limits deployment options and, consequently, how readily LLM-based solutions can be adopted. Fortunately, machine learning researchers are devising a growing range of solutions to meet the challenge of growing model sizes – with one of the most prominent being quantization. In this guide, we explore the concept of quant

A Guide to Quantization in LLMs by Kartik Talamadupula | 02-21-2024 The capabilities of Large Language Models (LLMs) have grown in leaps and bounds in recent years, making them more user-friendly and applicable in a growing number of use cases. However, as LLMs have increased in intelligence and complexity, the number of parameters, or weights and activations, i.e., its capacity to learn from and process data, has also grown. For example, GPT-3.5 has around 175 billion parameters, while the current state-of-the-art GPT-4 has in excess of 1 trillion parameters. However, the larger an LLM, the m

saved by

related reading