Zhiyuan and other institutions released the multi-skill large model governance strategy of the LM-Cocktail model

With the development and implementation of large model technology, "model governance" has become a proposition that currently receives major attention. However, in practice, researchers often face multiple challenges.

On the one hand, in order to improve its performance on the target task, researchers will collect and construct target task data sets and fine-tune the large language model (LLM), but this method It usually results in a significant decrease in the performance of general tasks other than the target task, compromising the original general capabilities of LLM.

On the other hand, the number of models in the open source community is gradually increasing, and large model developers may also accumulate more and more models in multiple trainings, each model has its own Advantages, how to choose the appropriate model to perform the task or further fine-tune it becomes a problem.

Recently, the Information Retrieval and Knowledge Computing Group of Zhiyuan Research Institute released LM-Cocktail model governance strategy, aiming to provide large model developers with a low-cost and sustainable Ways to improve model performance: Calculate fusion weights through a small number of samples, and use model fusion technology to combine the advantages of the fine-tuned model and the original model to achieve efficient use of "model resources."

• ##Technical report: https://arxiv.org/abs/2311.13534
• Code: https://github.com/FlagOpen/FlagEmbedding/tree/master/LM_Cocktail

Model fusion technology can fuse multiple Each model improves the performance of a single model. Inspired by this, the LM-Cocktail strategy further calculates the importance of different models for the target task, gives different models different weights, and performs model fusion on this basis. While improving the performance on the target task, it remains universal Strong ability in tasks.

The LM-Cocktail strategy is similar to making a cocktail. It can combine the advantages and capabilities of each model and create a unique model by blending different models. "Versatile" model with multiple specialties

Method innovation

Specifically, LM-Cocktail can manually select model configurations Ratio, or input a small number of samples to automatically calculate the weighted weights to fuse existing models to generate a new model. This process does not require retraining the model and has models that can adapt to multiple structures, such as large language model Llama, semantic vector model BGE et al.

If developers lack label data for certain target tasks or lack computing resources for model fine-tuning, they can use the LM-Cocktail strategy to eliminate the step of model fine-tuning. You only need to construct a very small amount of data samples, and you can integrate existing large-scale language models in the open source community to prepare your own "LM cocktail"

As shown in the figure above, fine-tuning Llama on a specific target task can significantly improve the accuracy on the target task, but harms the general ability on other tasks. Adopting LM-Cocktail can solve this problem.

The core of LM-Cocktail is to fuse the fine-tuned model with the parameters of multiple other models, integrating the advantages of multiple models, while improving the accuracy of the target task. Maintain general ability on other tasks. The specific form is as follows: given a target task, a basic model, and a model obtained by fine-tuning the basic model on the task, while collecting models from the open source community or previously trained models to form a collection. Calculate the fusion weight of each model through a small number of samples on the target task, and perform a weighted sum of the parameters of these models to obtain a new model (for the specific process, please refer to the paper or open source code). If there are no other models in the open source community, the basic model and the fine-tuned model can also be directly integrated to improve the performance of downstream tasks without reducing general capabilities.

In actual application scenarios, due to data and resource limitations, users may not be able to fine-tune downstream tasks, that is, there is no model that has been fine-tuned for the target task. In this case, users can construct a very small amount of data samples and integrate existing large language models in the community to generate a model for new tasks and improve the accuracy of the target task without training the model.

Experimental results

1. Flexible fine-tuning to maintain general capabilities

As you can see from the above figure, after fine-tuning on a certain target task, the fine-tuned model greatly improves the accuracy on that task, but other Accuracy decreased on general tasks. For example, after fine-tuning on the AG News training set, Llama's accuracy on the AG News test set increased from 40.80% to 94.42%, but its accuracy on other tasks dropped from 46.80% to 38.58%.

However, by simply fusing the parameters of the fine-tuned model and the original model, a competitive performance of 94.46% is achieved on the target task, which is comparable to the fine-tuned model, while achieving a competitive performance of 94.46% on other tasks. The accuracy is 47.73%, which is even slightly better than the performance of the original model. Under certain tasks, such as Helleswag, the fused model can even surpass the fine-tuned model on this fine-tuning task, and surpass the original general model on other tasks. That is, while inheriting the advantages of the fine-tuned model and the original model, it surpasses got them. It can be seen that calculating the fusion ratio through LM-Cocktail and further integrating other fine-tuned models can further improve the general performance on other tasks while ensuring the accuracy of the target task.

2. Mix existing models to handle new tasks

Rewrite The following content: The chart shows the target task of the language model MMLU

The rewritten content: Picture: The target task of the vector model is to retrieve (information Retrieval)

Fine-tuning the model requires a large amount of data and a large amount of computing resources, especially fine-tuning large language models, which may not be possible in actual situations. When the target task cannot be fine-tuned, LM-Cocktail can achieve new capabilities by mixing existing models (from the open source community or its own historical training accumulation).

By giving only 5 pieces of sample data, LM-Cocktail automatically calculates the fusion weights, filters existing models and then fuses them to obtain a new model without using a large amount of data. Conduct training. Experiments have found that the generated new model can achieve higher accuracy on new tasks. For example, for Llama, LM-Cocktail is used to fuse 10 existing models (the training tasks of which are not related to the MMLU list), which can achieve significant improvements, and is higher than the Llama model that uses 5 sample data for context learning.

Please try LM-Cocktail, we welcome your feedback and suggestions via GitHub issue: https://github.com/FlagOpen/FlagEmbedding/tree/master/LM_Cocktail

The above is the detailed content of Zhiyuan and other institutions released the multi-skill large model governance strategy of the LM-Cocktail model. For more information, please follow other related articles on the PHP Chinese website!