Towards 1-bit Machine Learning Models

Recent works on extreme low-bit quantization such as BitNet and 1.58 bit have attracted a lot of attention in the machine learning community. The main idea is that matrix multiplication with quantized weights can be implemented without multiplications, which can potentially be a game-changer in terms of compute efficiency of large machine learning models. We build on similar themes but aim to explore whether we can directly quantize pre-trained models with extreme settings, including binary weights (0s and 1s). Existing works are designed for training models from scratch. However, a vast array of excellent pre-trained open-source models are already available, such as Llama2. Moreover, training from scratch is an expensive task in terms of both computation and data, making these approaches less accessible for the open-source community. In this blog post, we delve into the details of extreme low-bit (2 and 1-bit) quantization of pre-trained models using HQQ+. HQQ+ is the adaptation of HQ

Recent works on extreme low-bit quantization such as BitNet and 1.58 bit have attracted a lot of attention in the machine learning community. The main idea is that matrix multiplication with quantized weights can be implemented without multiplications, which can potentially be a game-changer in terms of compute efficiency of large machine learning models. We build on similar themes but aim to explore whether we can directly quantize pre-trained models with extreme settings, including binary weights (0s and 1s). Existing works are designed for training models from scratch. However, a vast array