Ten lines of code are comparable to RLHF and use social game data to train a social alignment model

Making the behavior of language models consistent with human social values ​​is an important part of current language model development. The corresponding training is also called value alignment.

The current mainstream solution is RLHF (Reinforcenment Learning from Human Feedback) used by ChatGPT, which is reinforcement learning based on human feedback. This solution first trains a reward model (value model) as a proxy for human judgment. The agent model provides rewards as supervision signals to the generative language model during the reinforcement learning phase.

This method has the following pain points:

1. The rewards generated by the agent model can easily be destroyed or tampered with.

#2. During the training process, the agent model needs to continuously interact with the generative model, and this process may be very time-consuming and inefficient. In order to ensure high-quality supervision signals, the agent model should not be smaller than the generative model, which means that during the reinforcement learning optimization process, at least two larger models need to alternately perform inference (judgment of rewards) and parameter updating (generative model parameter optimization). Such a setting may be very inconvenient in large-scale distributed training.

#3. The value model itself has no obvious correspondence with the human thinking model. We do not have a separate scoring model in mind, and in fact it is very difficult to maintain a fixed scoring standard for a long time. Instead, much of the value judgment we form as we grow comes from daily social interactions—by analyzing different social responses to similar situations, we come to realize what is encouraged and what is not. These experiences and consensus gradually accumulated through a large amount of "socialization-feedback-improvement" have become the common value judgments of human society.

A recent study from Dartmouth, Stanford, Google DeepMind and other institutions shows that using high-quality data constructed by social games combined with simple and efficient alignment algorithms may be the only way to achieve this. The key to alignment.

• ## Article address: https://arxiv.org/pdf/2305.16960.pdf
• Code address: https://github.com/agi-templar/Stable-Alignment
• Model download (including base, SFT, and alignment models): https://huggingface.co/agi-css

The author proposes an alignment method trained on multi-agent game data. The basic idea can be understood as transferring the online interaction of the reward model and the generative model in the training phase to the offline interaction between a large number of autonomous agents in the game (high sampling rate, previewing the game in advance). The game environment runs independently of training and can be massively parallelized. Supervisory signals move from being dependent on the performance of the agent's reward model to being dependent on the collective intelligence of a large number of autonomous agents.

For this purpose, the author designed a virtual social model, called Sandbox. The sandbox is a world composed of grid points, and each grid point is a social agent. The social body has a memory system that is used to store various information such as questions, answers, feedback, etc. for each interaction. Every time the social group responds to a question, it must first retrieve and return the N historical questions and answers most relevant to the question from the memory system as a contextual reference for this reply. Through this design, the position of the social body can be continuously updated in multiple rounds of interaction, and the updated position can maintain a certain continuity with the past. Each social group has a different default position in the initialization phase.

##Convert game data into alignment data

In the experiment, the author used a 10x10 grid sandbox (a total of 100 social groups) to conduct social simulation, and formulated a social rule (the so-called Sandbox Rule): all social groups must make themselves aware of the problem The answers are more socially aligned to leave a good impression on other social groups. In addition, the sandbox also deployed observers without memory to score the responses of social groups before and after each social interaction. Scoring is based on two dimensions: alignment and engagement.

Simulated human society in a sandbox using different models

The author used the Sandbox to test language models of different sizes and different training stages. Overall, models trained with alignment (so-called “aligned models”), such as davinci-003, GPT-4, and ChatGPT, can generate socially normative responses in fewer interaction rounds. In other words, the significance of alignment training is to make the model safer in "out-of-the-box" scenarios without the need for special rounds of dialogue guidance. The model without alignment training not only requires more interactions to achieve the overall optimal response of alignment and engagement, but also the upper limit of this overall optimal is significantly lower than the aligned model.

The author also proposes a simple and easy alignment algorithm called Stable Alignment for Learn alignment from historical data in the sandbox. The stable alignment algorithm performs score-modulated contrastive learning in each mini-batch - the lower the score of the reply, the larger the boundary value of the contrastive learning will be set - in other words, stable alignment By continuously sampling small batches of data, the model is encouraged to generate responses that are closer to high-scoring responses and less close to low-scoring responses. Stable alignment eventually converges to the SFT loss. The authors also discuss the differences between stable alignment and SFT, RLHF.

The author particularly emphasizes the data from Sandbox games. Due to the setting of the mechanism, a large amount of it is included through revision ( revision) and become data that conforms to social values. The author proves through ablation experiments that this large amount of data with step-by-step improvement is the key to stable training.

##The author also aligns the algorithm performance with the current mainstream Performance comparisons were made with training stability, proving that stable alignment is not only more stable than reward modeling, but also comparable to RLHF in general performance and alignment performance (since ChatGPT uses undisclosed models, data and algorithms, it is only for reference ).

Instance generation results:

##Please refer to the paper for more details.

The above is the detailed content of Ten lines of code are comparable to RLHF and use social game data to train a social alignment model. For more information, please follow other related articles on the PHP Chinese website!