To run the ChatGPT volume model, you only need a GPU from now on: here is a method to accelerate it by a hundred times.

Computational cost is one of the major challenges that people face when building large models such as ChatGPT.

According to statistics, the evolution from GPT to GPT-3 is also a process of growth in model size - the number of parameters has increased from 117 million to 175 billion, and the amount of pre-training data has increased from 5GB. To 45TB, the cost of one GPT-3 training is US$4.6 million, and the total training cost reaches US$12 million.

In addition to training, inference is also expensive. Some people estimate that the computing power cost of OpenAI running ChatGPT is US$100,000 per day.

While developing technology to allow large models to master more capabilities, some people are also trying to reduce the computing resources required for AI. Recently, a technology called FlexGen has gained people's attention because of "an RTX 3090 running the ChatGPT volume model".

Although the large model accelerated by FlexGen still looks very slow - 1 token per second when running a 175 billion parameter language model, what is impressive is that it has The impossible became possible.

Traditionally, the high computational and memory requirements of large language model (LLM) inference necessitated the use of multiple high-end AI accelerators for training. This study explores how to reduce the requirements of LLM inference to a consumer-grade GPU and achieve practical performance.

Recently, new research from Stanford University, UC Berkeley, ETH Zurich, Yandex, Moscow State Higher School of Economics, Meta, Carnegie Mellon University and other institutions proposed FlexGen. is a high-throughput generation engine for running LLM with limited GPU memory.

By aggregating memory and computation from GPU, CPU and disk, FlexGen can be flexibly configured under various hardware resource constraints. Through a linear programming optimizer, it searches for the best pattern for storing and accessing tensors, including weights, activations, and attention key/value (KV) caches. FlexGen further compresses the weights and KV cache to 4 bits with negligible accuracy loss. Compared to state-of-the-art offloading systems, FlexGen runs OPT-175B 100x faster on a single 16GB GPU and achieves real-world generation throughput of 1 token/s for the first time. FlexGen also comes with a pipelined parallel runtime to allow super-linear scaling in decoding if more distributed GPUs are available.

Currently, this technology has released the code and has obtained thousands of stars: ​​https://www.php.cn /link/ee715daa76f1b51d80343f45547be570​​

#Introduction

In recent years, large language models have shown excellent performance in a wide range of tasks. While LLM demonstrates unprecedented general intelligence, it also exposes people to unprecedented challenges when building. These models may have billions or even trillions of parameters, resulting in extremely high computational and memory requirements to run them. For example, GPT-175B (GPT-3) requires 325GB of memory just to store model weights. For this model to do inference, you need at least five Nvidia A100s (80GB) and a complex parallelism strategy.

Methods to reduce the resource requirements of LLM inference have been frequently discussed recently. These efforts are divided into three directions:

(1) Model compression to reduce the total memory footprint;

(2) Collaborative inference, Amortize costs through decentralization;

(3) Offloading to utilize CPU and disk memory.

These techniques significantly reduce the computational resource requirements for using LLM. However, models are often assumed to fit in GPU memory, and existing offloading-based systems still struggle to run 175 billion parameter-sized models with acceptable throughput using a single GPU.

In new research, the authors focus on effective offloading strategies for high-throughput generative inference. When the GPU memory is not enough, we need to offload it to secondary storage and perform calculations piece by piece through partial loading. On a typical machine, the memory hierarchy is divided into three levels, as shown in the figure below. High-level memory is fast but scarce, low-level memory is slow but abundant.

In FlexGen, the author does not pursue low latency, but targets throughput-oriented scenarios, which are popular in applications such as benchmarking, information extraction, and data sorting. Achieving low latency is inherently a challenge for offloading, but for throughput-oriented scenarios, the efficiency of offloading can be greatly improved. Figure 1 illustrates the latency-throughput tradeoff for three inference systems with offloading. With careful scheduling, I/O costs can be spread over large amounts of input and overlap with computation. In the study, the authors showed that a single consumer-grade GPU throughput-optimized T4 GPU is 4 times more efficient than 8 latency-optimized A100 GPUs on the cloud in terms of cost per unit of computing power.

Figure 1. OPT-175B (left) and OPT-30B (right) top Latency and throughput tradeoffs for three offloading-based systems. FlexGen achieves a new Pareto optimal boundary, increasing the maximum throughput of the OPT-175B by a factor of 100. Other systems were unable to further increase throughput due to insufficient memory.

Although there have been studies discussing the latency-throughput trade-off of offloading in the context of training, no one has yet used it to generate LLM inference, which is A very different process. Generative inference presents unique challenges due to the autoregressive nature of LLMs. In addition to storing all parameters, it requires sequential decoding and maintaining a large attention key/value cache (KV cache). Existing offload systems are unable to cope with these challenges, so they perform too much I/O and achieve throughput well below the capabilities of the hardware.

Designing a good offloading strategy for generative inference is challenging. First, there are three tensors in this process: weights, activations and KV cache. The policy should specify what, where, and when to uninstall on a three-level hierarchy. Second, the structure of batch-by-batch, per-token, and per-layer calculations forms a complex dependency graph that can be calculated in a variety of ways. The strategy should choose a schedule that minimizes the execution time. Together, these choices create a complex design space.

To this end, on the new method FlexGen, an offloading framework for LLM inference is proposed. FlexGen aggregates memory from GPU, CPU, and disk and schedules I/O operations efficiently. The authors also discuss possible compression methods and distributed pipeline parallelism.

The main contributions of this study are as follows:

#1. The author formally defines the search space of possible offloading strategies and uses cost models and A linear programming solver searches for the optimal strategy. Notably, the researchers demonstrated that the search space captures an almost I/O-optimal computation order with an I/O complexity within 2 times the optimal computation order. The search algorithm can be configured for a variety of hardware specifications and latency/throughput constraints, providing a way to smoothly navigate the trade-off space. Compared to existing strategies, the FlexGen solution unifies weights, activations, and KV cache placement, enabling larger batch sizes.

2. Research shows that the weights and KV cache of LLMs such as OPT-175B can be compressed to 4 bits without retraining/calibration and with negligible accuracy loss. This is achieved through fine-grained grouping quantization, which can significantly reduce I/O costs.

3. Demonstrate the efficiency of FlexGen by running OPT-175B on NVIDIA T4 GPU (16GB). On a single GPU, given the same latency requirements, uncompressed FlexGen can achieve 65x higher throughput compared to DeepSpeed ​​Zero-Inference (Aminabadi et al., 2022) and Hugging Face Accelerate (HuggingFace, 2022) The latter is currently the most advanced inference system based on offloading in the industry. If higher latency and compression are allowed, FlexGen can further increase throughput and achieve 100x improvements. FlexGen is the first system to achieve 1 token/s speed throughput for the OPT-175B using a single T4 GPU. FlexGen with pipelined parallelism achieves super-linear scaling in decoding given multiple distributed GPUs.

In the study, the authors also compared FlexGen and Petals as representatives of offloading and decentralized set inference methods. Results show that FlexGen with a single T4 GPU outperforms a decentralized Petal cluster with 12 T4 GPUs in terms of throughput, and in some cases even achieves lower latency.

Operating Mechanism

By aggregating memory and computation from GPU, CPU and disk, FlexGen can be flexibly configured under various hardware resource constraints. Through a linear programming optimizer, it searches for the best pattern for storing and accessing tensors, including weights, activations, and attention key/value (KV) caches. FlexGen further compresses the weights and KV cache to 4 bits with negligible accuracy loss.

One of the key ideas of FlexGen is the latency-throughput trade-off. Achieving low latency is inherently challenging for offloading methods, but for throughput-oriented scenarios, offloading efficiency can be greatly improved (see figure below). FlexGen utilizes block scheduling to reuse weights and overlap I/O with computations, as shown in Figure (b) below, while other baseline systems use inefficient row-by-row scheduling, as shown in Figure (a) below.

Currently, the next steps of the study’s authors include support for Apple’s M1 and M2 chips and support for Colab deployment .

FlexGen has quickly received thousands of stars on GitHub since its release, and is also very popular on social networks. People have expressed that this project is very promising. It seems that the obstacles to running high-performance large-scale language models are gradually being overcome. It is hoped that within this year, ChatGPT can be handled on a single machine.

Someone used this method to train a language model, and the results are as follows:

Although it has not been fed with a large amount of data and the AI ​​does not know specific knowledge, the logic of answering questions seems relatively clear. Perhaps we can see such NPCs in future games?

The above is the detailed content of To run the ChatGPT volume model, you only need a GPU from now on: here is a method to accelerate it by a hundred times.. For more information, please follow other related articles on the PHP Chinese website!