Anonymous papers come up with surprising ideas! This can actually be done to enhance the long text capabilities of large models

When it comes to improving the long text capabilities of large models, do you think of length extrapolation or context window expansion?

No, these consume too much hardware resources.

Let’s look at a wonderful new solution:

The essence of using KV cache is different from methods such as length extrapolation. It uses parameters of the modelTo store a large amount of contextual information.

The specific method is to build a temporary Lora module , so that it can only "stream update" during the long text generation process, that is Previously generated content is continuously used as input to serve as training data to ensure that knowledge is stored in the model parameters.

ThenOnce the inference is complete, throw it away to ensure that it does not have a long-term impact on the model parameters.

This method allows us to store context information at will without expanding the context window, save as much as we want.

Experiments have proven that this method:

• can significantly improve the quality of long text tasks of the model, achieving a 29.6% reduction in confusion, and the quality of long text translation (BLUE score) increased by 53.2%;
• is also compatible with and enhances most existing long text generation methods.
• The most important thing is that it can greatly reduce computing costs.

While ensuring a slight improvement in generation quality (perplexity reduced by 3.8%), the FLOPs required for reasoning are reduced by 70.5% and the delay is reduced by 51.5%!

For the specific situation, let’s open the paper and take a look.

Create a temporary Lora module and throw it away after use

The method is called Temp-Lora, and the architecture diagram is as follows:

The core is to gradually train the temporary Lora module on the previously generated text in an autoregressive manner.

This module is very adaptable and can be continuously adjusted, so it can have a deep understanding of contexts of different distances.

The specific algorithm is as follows:

During the generation process, tokens are generated block by block. Each time a block is generated, the latest L_xtoken is used as input X to generate subsequent tokens.

Once the number of generated tokens reaches the predefined block size Δ, start the training of the Temp-Lora module using the latest block, and then start the next block generation.

In the experiment, the author sets Δ L_x to W to make full use of the model’s context window size.

For the training of the Temp-Lora module, learning to generate new blocks without any conditions may not constitute an effective training goal and lead to serious overfitting.

To solve this problem, the author incorporates the L_T tags in front of each block into the training process, using them as input and the block as output.

Finally, the author also proposed a strategy called Cache Reuse (Cache Reuse) to achieve more efficient reasoning.

Generally speaking, after updating the Temp-Loramo module in the standard framework, we need to recalculate the KV status using the updated parameters.

Alternatively, reuse the existing cached KV state while using the updated model for subsequent text generation.

Specifically, we recalculate the KV state using the latest Temp-Lora module only when the model generates a maximum length (context window size W) .

Such a cache reuse method can speed up the generation without significantly affecting the generation quality.

That’s all the introduction to the Temp-Lora method. Let’s focus on the tests below.

The longer the text, the better the effect

The author tested the Temp- The Lora framework is evaluated and covers both types of long text tasks: generation and translation.

The test data set is a subset of the long text language modeling benchmark PG19, from which 40 books were randomly selected.

The other is a randomly sampled subset of the Guofeng data set from WMT 2023, which contains 20 Chinese online novels translated into English by professionals.

First let’s look at the results on PG19.

The table below shows the PPL (Perplexity, reflecting the uncertainty of the model for a given input) for various models on PG19 with and without the Temp-Lora module sex, the lower the better)Compare. Divide each document into segments ranging from 0-100K to 500K tokens.

It can be seen that the PPL of all models dropped significantly after Temp-Lora, and as the fragments become longer and longer, the impact of Temp-Lora is more obvious (1-100K only decreased by 3.6% , 500K reduced by 13.2%).

Therefore, we can simply conclude: The more text there is, the stronger the necessity to use Temp-Lora.

In addition we can also find that adjusting the block size from 1024 to 2048 and 4096 results in a slight increase in PPL.

This is not surprising, after all, the Temp-Lora module is trained on the data of the previous block.

This data mainly tells us that the choice of block size is a key trade-off between generation quality and computational efficiency (further analysis can be found in the paper) .

Finally, we can also find that cache reuse will not cause any performance loss.

The author said: This is very encouraging news.

The following are the results on the Guofeng data set.

It can be seen that Temp-Lora also has a significant impact on long text literary translation tasks.

Significant improvement in all metrics compared to the base model: PPL reduced by -29.6%, BLEU score (similarity of machine translated text to high-quality reference translation) improved 53.2%, and the COMET score (also a quality indicator) increased by 8.4%.

Finally, there is the exploration of computational efficiency and quality.

The author found through experiments that using the most "economical" Temp-Lora configuration (Δ=2K, W=4K) can While reducing PPL by 3.8%, it saves 70.5% of FLOPs and 51.5% of latency.

On the contrary, if we completely ignore the computational cost and use the most "luxurious" configuration (Δ=1K and W=24K) , It is also possible to achieve a 5.0% PPL reduction and an additional 17% FLOP and 19.6% latency.

Usage Suggestions

To summarize the above results, the author also gives three suggestions for the practical application of Temp-Lora:

1. For applications that require the highest level of long text generation , without changing any parameters, integrating Temp-Lora into the existing model can significantly improve performance at a relatively moderate cost.

2. For applications that value minimal latency or memory usage, the computational cost can be significantly reduced by reducing the input length and contextual information stored in Temp-Lora.

In this setting, we can use a fixed short window size (such as 2K or 4K) to handle almost infinitely long text (500K in the author's experiments ).

3. Finally, please note that Temp-Lora is useless in scenarios that do not contain a large amount of text, such as when the context in pre-training is smaller than the window size of the model.

The author comes from a confidential organization

It is worth mentioning that the author did not leave much source information for inventing such a simple and innovative method:

The name of the organization is directly signed "Secret Agency", the names of the three authors also only have full surnames.

#However, judging from the email information, it may be from schools such as City University of Hong Kong and Hong Kong Chinese Language School.

Finally, what do you think of this method?

Paper: https://www.php.cn/link/f74e95cf0ef6ccd85c791b5d351aa327

The above is the detailed content of Anonymous papers come up with surprising ideas! This can actually be done to enhance the long text capabilities of large models. For more information, please follow other related articles on the PHP Chinese website!