"Putting" a large 33 billion parameter model into a single consumer-grade GPU, speeding up 15% without sacrificing performance

The performance of pre-trained large language models (LLM) on specific tasks continues to improve. Subsequently, if the prompt instructions are appropriate, it can be better generalized to more tasks. Many people will This phenomenon is attributed to the increase in training data and parameters. However, recent trends show that researchers are focusing more on smaller models, but these models are trained on more data and are therefore easier to infer. use.

For example, LLaMA with a parameter size of 7B was trained on 1T tokens. Although the average performance is slightly lower than GPT-3, the parameter size is 1/25 of the latter. Not only that, but current compression technology can further compress these models, significantly reducing memory requirements while maintaining performance. With such improvements, well-performing models can be deployed on end-user devices such as laptops.

However, this faces another challenge, which is how to compress these models into a small enough size to fit these devices, while taking into account the generation quality. Research shows that while compressed models generate answers with acceptable accuracy, existing 3-4-bit quantization techniques still degrade accuracy. Since LLM generation is performed sequentially and relies on previously generated tokens, small relative errors accumulate and lead to severe output corruption. To ensure reliable quality, it is critical to design low bit-width quantization methods that do not degrade prediction performance compared to 16-bit models.

However, quantizing each parameter to 3-4 bits often results in moderate or even high accuracy loss, especially those 1-10B that are well suited for edge deployments Smaller model within parameter range.

In order to solve the accuracy problem, researchers from the University of Washington, ETH Zurich and other institutions proposed a new compression format and quantization technology SpQR (Sparse-Quantized Representation, sparse - quantified representation), achieving near-lossless compression of LLM across model scales for the first time while achieving similar compression levels to previous methods.

SpQR works by identifying and isolating anomalous weights that cause particularly large quantization errors, storing them with higher precision while compressing all other weights. To position 3-4, less than 1% perplexity relative accuracy loss is achieved in LLaMA and Falcon LLMs. This allows a 33B parameter LLM to be run on a single 24GB consumer GPU without any performance degradation while being 15% faster.

The SpQR algorithm is efficient and can both encode weights into other formats and decode them efficiently at runtime. Specifically, this research provides SpQR with an efficient GPU inference algorithm that can perform inference faster than 16-bit baseline models while achieving over 4x memory compression gains.

• Paper address: https://arxiv.org/pdf/2306.03078.pdf
• Project address: https://github.com/Vahe1994/SpQR

##Method

This research proposes a new format for hybrid sparse quantization - Sparse Quantization Representation (SpQR), which can compress accurately pre-trained LLM to 3-4 bits per parameter while remaining nearly lossless.

Specifically, the study divided the entire process into two steps. The first step is outlier detection: the study first isolates the outlier weights and demonstrates that their quantization leads to high errors: outlier weights are maintained with high precision, while other weights are stored with low precision (e.g. in a 3-bit format). The study then implements a variant of grouped quantization with very small group sizes and shows that the quantization scale itself can be quantized into a 3-bit representation.

SpQR greatly reduces the memory footprint of LLM without sacrificing accuracy, while producing LLM 20%-30% faster compared to 16-bit inference.

In addition, the study found that the positions of sensitive weights in the weight matrix are not random, but have a specific structure. To highlight its structure during quantification, the study calculated the sensitivity of each weight and visualized these weight sensitivities for the LLaMA-65B model. Figure 2 below depicts the output projection of the last self-attention layer of LLaMA-65B.

The study made two changes to the quantification process: one to capture small sensitive weight groups; Used to capture individual outliers. Figure 3 below shows the overall architecture of SpQR:

The following table shows the SpQR quantification algorithm. The code fragment on the left describes the entire process, the code snippet on the right contains subroutines for secondary quantification and finding outliers:

Experiment

This study will SpQR is compared with two other quantization schemes: GPTQ, RTN (rounding-to-nearest), and two metrics are used to evaluate the performance of the quantization model. The first is the measurement of perplexity, using data sets including WikiText2, Penn Treebank and C4; the second is the zero-sample accuracy on five tasks: WinoGrande, PiQA, HellaSwag, ARC-easy, ARC-challenge.

Main results. Figure 1 results show that at similar model sizes, SpQR performs significantly better than GPTQ (and corresponding RTN), especially on smaller models. This improvement is due to SpQR achieving more compression while also reducing loss degradation.

Table 1, Table 2 The results show that for 4-bit quantization, the error of SpQR relative to the 16-bit baseline is halved compared to GPTQ.

##Table 3 reports the LLaMA-65B model Perplexity results on different data sets.

Finally, the study evaluates SpQR inference speed. This study compares a specially designed sparse matrix multiplication algorithm with the algorithm implemented in PyTorch (cuSPARSE), and the results are shown in Table 4. As you can see, although standard sparse matrix multiplication in PyTorch is not faster than 16-bit inference, the sparse matrix multiplication algorithm specially designed in this article can improve the speed by about 20-30%.

The above is the detailed content of "Putting" a large 33 billion parameter model into a single consumer-grade GPU, speeding up 15% without sacrificing performance. For more information, please follow other related articles on the PHP Chinese website!