Quantization

huggingface.co · 787 words · saved by 1 readers

GPTQ is a post-training quantization method to make the model smaller. It quantizes the layers by finding a compressed version of that weight, that will yield a minimum mean squared error like below 👇 Given a layer 𝑙 l with weight matrix 𝑊 𝑙 W l and layer input 𝑋 𝑙 X l , find quantized weight ℎ 𝑎 𝑡 𝑊 𝑙 hatW l : ( 𝑊 ^ 𝑙 ∗ = 𝑎 𝑟 𝑔 𝑚 𝑖 𝑛 𝑊 𝑙 ^ ∣ ∣ 𝑊 𝑙 𝑋 − 𝑊 ^ 𝑙 𝑋 ∣ ∣ 2 2 ) ( W ^ l ∗ =argmin W l ^ ∣∣W l X− W ^ l X∣∣ 2 2 ) TGI allows you to both run an already GPTQ quantized model (see available models here) or quantize a model of your choice using quantization script. You can run a quantized model by simply passing —quantize like below 👇 Note that TGI’s GPTQ implementation doesn’t use AutoGPTQ under the hood. However, models quantized using AutoGPTQ or Optimum can still be served by TGI. To quantize a given model using GPTQ with a calibration dataset, simply run This will create a new directory with the quantized files whi

Quantization · Hugging Face text-generation-inference documentation Quantization text-generation-inference 🏡 View all docs AWS Trainium & Inferentia Accelerate Argilla AutoTrain Bitsandbytes CLI Chat UI Dataset viewer Datasets Deploying on AWS Diffusers Distilabel Evaluate Google Cloud Google TPUs Gradio Hub Hub Python Library Huggingface.js Inference Endpoints (dedicated) Inference Providers Kernels LeRobot Leaderboards Lighteval Microsoft Azure OpenEnv Optimum PEFT Reachy Mini Safetensors Sentence Transformers TRL Tasks Text Embeddings Inference Text Generation Inference Tokenizers Trackio

Quantization

related reading