Targeting the intelligent society, MetaGPT joins hands with Jürgen Schmidhuber team

In the past few months, MetaGPT [1]’s Agents software company examples have been impressive. It quickly gained 30k stars on GitHub and dozens of global Professional media and big V reports. But agent software companies are just a microcosm of Agent Society. An intelligent society may have software companies, e-commerce companies, and game companies, as well as a large number of independent agents to provide productivity. Jürgen Schmidhuber, the father of modern artificial intelligence, also strongly recognizes the concept of an intelligent society. He and his team have made significant contributions to MetaGPT and are included in the list of MetaGPT authors.

As early as 1986, Marvin Minsky led an ideological revolution in the field of artificial intelligence with his work "Society of Mind (SOM)" [2]. He proposed a very original theory: the mind does not need to be composed of individual intelligent components, but rather a complex system composed of the interaction of a series of simple components. It is this collection that gave birth to what we know. Intelligence and consciousness. This concept has had an immeasurable and far-reaching impact on the construction of autonomous agents and their subsequent development.

With the leap forward in artificial intelligence technology to 2023, we can now imagine how if each tiny component itself possesses a certain degree of intelligence, how they will interact and what level of collective intelligence they will produce. Among the research papers on Natural Language Agent Society (NLSOM, Language Agent Society) in the first half of 2023 [3], from well-known research institutions such as King Abdullah University of Science and Technology, Swiss Artificial Intelligence Laboratory, Oxford University and ETH Zurich scientists jointly explored the possibility of intelligent agent communities.

They proposed to build a language-driven community of agents that can collaboratively complete tasks that a single agent cannot or is difficult to complete independently. The study proposes a series of experimental ideas that are more than just proofs of concept. They are seen as a precursor to a society containing trillions of intelligent agents, which may also include human members.

https://arxiv.org/pdf/2305.17066.pdf

At the 2023 CogX Festival, Jürgen showed the audience his vision for large-scale Insights from Language Models (LLMs). When discussing topics related to agents, he mentioned various ways to build self-improving systems, including Universal Turing Machines [4] and Gödel machines [5]. He pointed out that the current large language model provides us with a new way of thinking - by using a universal symbolic language (such as natural language or programming code) as an interface to connect different models. These models can communicate with other language models to build a natural language society of mind (NLSOM) paradigm.

Professor Jürgen Schmidhuber is Scientific Director of the Swiss Artificial Intelligence Laboratory (IDSIA) and Director of the Artificial Intelligence Center at King Abdullah University of Science and Technology (AI Initiative, KAUST) . His work has a profound impact on important artificial intelligence directions such as reinforcement learning (Reinforcement Learning), meta learning (Meta Learning), and neural networks (Neural Network).

So far, Professor Schmidhuber’s Google Scholar citations are 210,000, of which the long short-term memory (LSTM) paper as a co-inventor has been cited 90,000 times. At the age of 15, he hoped to develop an artificial intelligence that was smarter and capable of self-improvement so that he could retire. Two of the four people who founded DeepMind, as well as their first AI Ph.D. recruit, came from Jürgen Schmidhuber’s lab.

In the society Jürgen envisions, all communication is transparent and easily interpretable. He mentioned a concept called "Mindstorm", that is, when a problem is given, this natural language mind society can cooperate to solve the problem.

In this process, each member of society may have different ideas and perspectives, and they will collect and integrate these different ideas to make collective decisions.

This method is particularly suitable for solving problems that cannot be effectively solved by a single agent. Jürgen further explained that such problems can be of a programming nature, such as using the Python language to solve a specific programming problem. Through this synergy, the collective intelligence of the intelligent agent society will be able to achieve solutions that transcend individual capabilities.

This iteration of the MetaGPT project received direct guidance from Jürgen, and his team also provided a lot of support in coding, writing, and engineering.

Next, this article will analyze the updated content of the MetaGPT paper in detail so that readers can understand its details more deeply.

1. Paper and Framework Update

Paper Section 3.1 Update: Expounds the concept of role specialization design and role division of labor in the MetaGPT framework, and illustrates the role of a single agent in Behavioral patterns in MetaGPT and organization under SOPs.

Paper section 3.2 update: Introducing the communication mechanism in the MetaGPT framework, including structured communication interface design and publish-subscribe mechanism.

Update to Section 3.3 of the paper: An executable feedback mechanism is introduced, which is a mechanism for continuous iteration and self-correction during code execution.

Fig.2. Example of communication protocol (left) and example of iterative programming with on-the-fly executable feedback (right). Left: Agents use a shared message pool to publish structured messages. They can also subscribe to relevant messages based on their configuration. Right picture: After generating the initial code, the engineer agent can execute the code and check whether errors are reported during operation. If an error occurs, the Agent will check the execution results and compare them with PRD, system design and code files to rewrite and optimize the code.

1.1. Agent communication protocol

Currently, most multi-agent agents complete collaboration through dialogue based on natural language, but this is not suitable for solving specific tasks. The best way.

Natural language output without constraints and specific requirements may lead to distortion of information content or shift in semantic focus.

Therefore, structured communication content and interface forms help agents to quickly and accurately understand task requirements, and are also conducive to maximizing the retention of information content. Referring to the role requirements for different positions in human SOPs, we set output specifications for each role that are in line with human experts in the corresponding position, requiring the agent to convert the original natural language information into a more structured expression (as shown in the figure below). Such as data structure, API design and sequence diagram.

Fig.3 MetaGPT software development process diagram shows that structured SOPs can bring better results. For a more detailed demonstration, see Appendix B

In subsequent experiments, we compared the actual solution effects of MetaGPT and ChatDev (using a chat-form communication and collaboration mechanism) to perform this complex task of software development. The results illustrate the structure. Specialized communication interface design can bring significant effects to intelligent agent collaboration.

Publish-Subscribe Mechanism

In the communication process of multi-agent, relying only on the 1v1 single-point communication method will not only increase the complexity of the communication topology, but also lead to collaboration The low efficiency will also sharply increase development costs. Therefore, we implemented a shared message pool and interest-based subscription within the framework through the [publish-subscribe] messaging mechanism.

Specifically, the environment provides a shared message pool from which agents can obtain information directly without asking other agents one by one. At the same time, the agent can filter and filter messages based on its own interests/concerned behaviors, thereby reducing message/memory overload. As shown in Figure 3, the architect mainly pays attention to the product manager's PRD document output, but pays less attention to the test engineer's document.

1.2. Executable iterative feedback design

Debugging and execution feedback play an important role in daily programming tasks. However, existing methods often lack self-correction mechanisms and only conduct code feasibility assessment through code review and review mechanisms. To further reduce the hallucination problem of LLM in generating code, we introduce an executable feedback mechanism to iteratively improve the code. Through automatic code execution test result feedback, code feasibility assessment and judgment are carried out, and LLM is promoted to carry out self-iteration and optimization. As shown in Figure 2, engineers can continuously update the code based on the code execution results and iteratively test until the test passes or exits after a maximum of N retries.

2. Experiment update

In the experimental part, we have added an exploratory experiment on the effect of introducing a multi-agent framework into SOPs, and code brought by executable iterative feedback Quality improvement experiments. On the dataset:

1. Effect evaluation on code quality: We use two public benchmark data sets: HumanEval and MBPP.
   1) HumanEval includes 164 handwritten programming tasks. These tasks include functional specifications, descriptions, reference code, and tests.
   2) MBPP contains 427 Python tasks. These tasks cover core concepts and standard library functionality and include instructions, reference code, and automated tests.
2. We propose SoftwareDev, a benchmark dataset for more challenging software development tasks: Our SoftwareDev dataset collects 70 representative software development task instances, each with its own task prompt (See Table 5 of the paper). The range of these tasks is diverse (see Figure 5 of the paper), such as mini-games, image processing algorithms, data visualization, etc. They provide a powerful testing platform for real development tasks. Unlike previous datasets, SoftwareDev focuses on engineering aspects. In the comparison, we randomly selected seven representative tasks for evaluation.

2.1. Executable iterative feedback design

Figure 4 shows that MetaGPT performs well in both the HumanEval and MBPP benchmarks. Outperforms all previous methods. When MetaGPT (using GPT-4 as the base model), its Pass@1 on the HumanEval benchmark is significantly improved compared to GPT-4. It achieved 85.9% and 87.7% in these two public benchmarks (considering the experimental cost, the numerical results of some models are directly used the results provided by Dong et al. (2023). [6]).

##Figure 4: Pass rates on the MBPP and HumanEval with a single attempt.

2.2. Software development task data set & evaluation indicators

For SoftwareDev, we prioritize the actual usability of build projects and evaluate performance through manual evaluation (A, E) or statistical analysis (B, C, D). We demonstrate MetaGPT through visual examples The ability to generate independent software (Figure 5 of the paper). Additional experiments and analysis can be found in Appendix C of the paper:

(A) Executability: This metric rates the generated code from 1 (failed/non-functional) to 4 (defect-free). 1 means non-functional, 2 means working but not perfect, 3 means nearly perfect, and 4 means no defects.

(B) Cost: The cost evaluation here includes (1) project running time (2) Token consumption and (3) actual costs.

(C) Code statistics: including (1) the number of code files (2) the average number of lines of code per file and (3) the total number of lines of code.

(D) Production efficiency: The basic definition is the Token usage divided by the number of lines of code, that is, the Token consumed by each line of code. The smaller the value, the higher the code production efficiency.

(E) Manual revision cost: Quantified by the number of revision rounds required to ensure that the code runs smoothly, which represents the frequency of manual intervention, such as debugging or importing dependencies and other revisions.

2.3, SOPs vs ChatChain

In the scenario of solving specific tasks, in order to explore the effect of SOPs on multi-agent collaboration ,We selected ChatDev, an agent framework that supports ,software development tasks in open source work, as the ,experimental comparison object. ChatDev is a framework for agent organization and collaboration based on the role division of ChatChain and software development waterfall flow. We selected 7 tasks from SoftwareDev for comparison and compared the above-mentioned relevant indicators to illustrate the differences.

As shown in Table 1 of the paper, MetaGPT outperforms ChatDev in almost all metrics on the challenging SoftwareDev dataset.

For example: MetaGPT scored 3.75 for executability, which is very close to 4 (perfect). Additionally, it takes significantly less time (503 seconds) than ChatDev.

It is also significantly better than ChatDev in terms of code statistics and manual modification costs. Although MetaGPT requires more tokens (24,613 or 31,255 compared to ChatDev’s 19,292), it only requires 126.5/124.3 Tokens to generate a line of code. In comparison, ChatDev uses 248.9 Tokens.

These results highlight the advantages of SOPs in multi-agent collaboration.

#3. Acknowledgments

Thanks to Sarah Salhi, executive secretary of KAUST AI Center, postdoctoral fellow Wang Yuhui, and doctoral student Wang Wenyi for this The paper provides suggestions and help.

[1] https://arxiv.org/pdf/2308.00352.pdf

[2] https://en.wikipedia.org/wiki/Society_of_Mind

[3] https://arxiv.org/pdf/2305.17066.pdf

[4] https://en.wikipedia.org/wiki/Universal_Turing_machine

[5] https://en.wikipedia.org/wiki/Gödel_machine

[6] https://arxiv.org/abs/2304.07590

The above is the detailed content of Targeting the intelligent society, MetaGPT joins hands with Jürgen Schmidhuber team. For more information, please follow other related articles on the PHP Chinese website!