Winning allies and understanding people's hearts, the latest Meta agent is a master negotiator

Gaming has long been a proving ground for advances in AI—from Deep Blue’s victory over chess grandmaster Garry Kasparov, to AlphaGo’s mastery of Go beyond humans, to Pluribus beating the best players at poker. . But a truly useful, omnipotent agent can't just play a board game and move chess pieces around. One can’t help but ask: Can we build a more effective and flexible agent that can use language to negotiate, persuade, and work with people to achieve strategic goals like humans?

In the history of games, there is a classic tabletop game Diplomacy. When many people see the game for the first time, they will be shocked by its map-style board. Think of it as a complex war game. In fact, this is not the case. This is a game that requires mobilizing language to win allies. It involves decision-making and negotiation. There is a lot of communication between players. The key to winning the game lies in the interaction between people.

Now Meta has launched a challenge to this game. The intelligent agent they built, CICERO, has become the first AI to reach human level in Diplomacy. CICERO demonstrated this on the online version webDiplomacy.net, where CICERO scored on average more than double that of human players and ranked in the top 10% of participants who had played more than one game.

• Paper address: https://www.science.org/doi/10.1126/ science.ade9097
• Homepage address: https://ai.facebook.com/research/cicero/diplomacy/

For ten years, Diplomacy has been regarded as an insurmountable challenge in the field of AI because the game requires players to understand the motivations and perspectives of others, make complex plans, adjust strategies, and use natural language to reach agreements with others on this basis. , persuade others to form partnerships and alliances, etc. These are still difficult for agents, and CICERO is still very effective in using natural language to negotiate with Diplomacy players.

Unlike chess and Go, Diplomacy is a game about people, not pieces. If an agent cannot tell whether an opponent is bluffing or actually sabotaging, it will quickly lose the game. Likewise, if an agent can't communicate like a human, show empathy, build relationships with others, and talk about the game - it won't find other players willing to cooperate with it.

Meta’s research combines strategic reasoning (such as AlphaGo, Pluribus) with natural language processing (such as GPT-3, BlenderBot 3, LaMDA, OPT-175B) was combined. For example, late in the game CICERO deduces that it will need the support of a specific player, and CICERO then develops a strategy to win that person's favor.

How to build CICERO

The core of CICERO is a controllable dialogue model and a strategic reasoning engine. At every point in the game, CICERO looks at the game board and its conversation history and models what other players might do. A plan is then developed to control the language model, communicate its plans to other players, and suggest reasonable actions to other players that coordinate well with them.

Controllable dialogue​

In order to build a controllable dialogue model, Meta starts from a controllable dialogue model with 2.7 billion parameters. We started with a BART-like language model, pre-trained on text from the Internet, and fine-tuned on over 40,000 human games on webDiplomacy.net.

The implementation process is mainly divided into the following steps:

#Step 1: Based on the board state and the current dialogue, CICERO will make a decision for each person What gives an initial prediction.

Step 2: CICERO iteratively improves the initial forecast and then uses the improved forecast to form an intention for itself and its partners.

#Step 3: Generate multiple candidate messages based on board state, dialogue and intent.

​

Step 4: Filter candidate messages, maximize the value, and ensure that the intentions of each other are consistent.

The researchers used some filtering mechanisms to further improve the quality of the dialogue, such as using trained classifiers to distinguish between humans and models Generated text to ensure - that dialogue makes sense, is consistent with the current game state and previous information, and is strategically sound.

Conversation-aware strategy and planning

In games involving cooperation, agents need to learn to simulate humans in reality What do people actually do in life, rather than treating humans as machines with agents dictating what they should do. Meta therefore hopes that the plans developed by CICERO will be consistent with the dialogue with other actors.

​

The classic method of human modeling is supervised learning, which uses labeled data (such as human players in past games action database) to train the agent. However, relying purely on supervised learning to choose actions based on past conversations results in an agent that is relatively weak and easily exploited. For example, a player could tell the agent "I'm glad we agreed that you will move your troops away from Paris!" Since similar information only appears in the training data when an agreement is reached, the agent may actually moved its troops away from Paris, even though doing so was a clear strategic mistake.

To solve this problem, CICERO runs an iterative planning algorithm to balance the consistency and reasonableness of the conversation. The agent first predicts each player's strategy for the current turn based on its conversations with other players, and also predicts what other players think the agent's strategy will be. It will then run a planning algorithm called "piKL", which iteratively improves these predictions by trying to choose new strategies with higher expected values ​​given the strategies predicted by other players, while also trying to make the new predictions close to the original Strategic Forecasting. The researchers found that piKL can better simulate human games and bring better strategies to the agent than pure supervised learning.

​​

Generate natural, purposeful dialogue

In In Diplomacy, how players talk to each other is even more important than how they move their pieces. CICERO is able to communicate clearly and persuasively when strategizing with other players. For example, in one demo, CICERO asked one player to immediately support a certain part of the board, while putting pressure on another player to consider an alliance later in the game.

In the exchange, CICERO attempts to execute its strategy by proposing actions to three different players. In the second conversation, the agent is able to tell the other players why they should cooperate and how it will benefit both parties. In this third dialogue, CICERO is both soliciting information and laying the groundwork for future action.

Weaknesses

It must be acknowledged that CICERO can sometimes produce inconsistent dialogue that undermines its goals. In the example below, CICERO plays Austria, but the agent asks Italy to move to Venice, contradicting its first message.

Let "Diplomacy" advance the sandbox of human-AI interaction

In an environment that involves both cooperation The emergence of goal-oriented dialogue systems in games that also involve competition poses important social and technical challenges in aligning AI with human intentions and goals. Diplomacy provides a particularly interesting setting for studying this problem, because playing the game requires wrestling with conflicting goals and translating these complex goals into natural language. As a simple example, a player may choose to compromise on short-term benefits to maintain a relationship with an ally because that ally may help them get into a better position next turn.

While Meta has made significant progress in this work, the ability to powerfully combine language models with concrete intents, and the technical (and normative) challenges of determining those intents, remain is an important question. By open sourcing the CICERO code, Meta hopes that AI researchers can continue to build on this work in a responsible way. The team said: “By using conversation models for zero-shot classification, we have taken early steps in detecting and removing harmful information in this new field. We hope that “Diplomacy” can serve as a safe sandbox to advance human-AI interaction. Research. 》

FUTURE DIRECTIONS

While CICERO is currently only capable of playing Diplomacy games, the technology behind this achievement is relevant to many real-world applications. For example, communication barriers between humans and AI-driven agents can be alleviated by controlling natural language generation through planning and RL.

For example, today’s AI assistants are great at answering simple questions, like telling you the weather, but what if they could sustain long-term conversations with the goal of teaching you a new skill?

Also, imagine a video game where NPCs can plan and talk like humans—understanding your motivations and tailoring dialogue accordingly to help you complete your quest to storm a castle.

These "dreams" may become reality in the future.

The above is the detailed content of Winning allies and understanding people's hearts, the latest Meta agent is a master negotiator. For more information, please follow other related articles on the PHP Chinese website!