Tsinghua University and other open source "tool learning benchmark" ToolBench, fine-tuning model ToolLLaMA performance surpasses ChatGPT

Human beings have the ability to create and utilize tools, allowing us to break through the limitations of the body and explore a wider world.

The basic model of artificial intelligence is similar. If you only rely on the weights obtained in the training stage, the usage scenarios will be very limited. However, the recently proposed tool learning combines specialized tools in specific fields with large-scale The combination of basic models can achieve higher efficiency and performance.

However, the current research on tool learning is not in-depth enough, and there is a lack of relevant open source data and code.

Recently, OpenBMB (Open Lab for Big Model Base), an open source community supported by Tsinghua University Natural Language Processing Laboratory and others, released the ToolBench project, which can help developers build open source, large-scale, High-quality instruction tuning data that facilitates the construction of large language models with the ability to use common tools.

##Repository link: https://github.com/OpenBMB/ToolBench

The ToolBench warehouse provides relevant data sets, training and evaluation scripts, and the functional model ToolLLaMA fine-tuned on ToolBench. The specific features are:

1. Supports single tool and multiple tools Tool solution

The single tool setting follows the LangChain prompt style, and the multi-tool setting follows the AutoGPT prompt style.

2. The model reply not only includes the final answer, but also includes the model’s thinking chain process, tool execution and tool execution results

3. Support Real-world level complexity, supporting multi-step tool calls

4. Rich API that can be used for real-world scenarios such as weather information, search, stock updates, and PowerPoint automation

5. All data is automatically generated by the OpenAI API and filtered by the development team. The data creation process is easily scalable

However It should be noted that the data released so far is not final, and researchers are still post-processing the data to improve data quality and increase the coverage of real-world tools.

ToolBench

The general idea of ​​ToolBench is to train large language models in supervised data based on BMTools.

##The warehouse contains 9,800 pieces of data obtained from 312,000 real API calls, covering single Tool scenario and multi-tool scenario, the following is the statistical information of single tool.

Each row of data is a json dict, including the prompt template for data creation, Human instructions (queries) for tool usage, intermediate thought/tool ​​execution loops, and final answers.

Tool Descrition:BMTools Tool_name: translationTool action: get_translationaction_input: {"text": target texts, "tgt_lang": target language}Generated Data:{"prompt": "Answer the following questions as best you can. Specifically, you have access to the following APIs:\n\nget_translation: . Your input should be a json (args json schema): {{\"text\" : string, \"tgt_lang\" : string, }} The Action to trigger this API should be get_translation and the input parameters should be a json dict string. Pay attention to the type of parameters.\n\nUse the following format:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction: the action to take, should be one of [get_translation]\nAction Input: the input to the action\nObservation: the result of the action\n... (this Thought/Action/Action Input/Observation can repeat N times, max 7 times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nBegin! Remember: (1) Follow the format, i.e,\nThought:\nAction:\nAction Input:\nObservation:\nFinal Answer:\n (2) Provide as much as useful information in your Final Answer. (3) Do not make up anything, and if your Observation has no link, DO NOT hallucihate one. (4) If you have enough information and want to stop the process, please use \nThought: I have got enough information\nFinal Answer: **your response. \n The Action: MUST be one of the following:get_translation\nQuestion: {input}\n Agent scratchpad (history actions):\n {agent_scratchpad}","query": "My intention is to convert the data provided in ما هي الأقسام الثلاثة للقوات المسلحة؟ into Arabic(ara).\n","chains": [{"thought": "I need to use the get_translation API to convert the text into Arabic.","action": "get_translation","action_input": "{\"text\": \"What are the three branches of the military?\", \"tgt_lang\": \"ara\"}","observation": "\"ما هي الفروع الثلاثة للجيش ؟\""}],"answer": "The translation of \"What are the three branches of the military?\" into Arabic is \"ما هي الفروع الثلاثة للجيش ؟\"."}

Model experiment

Machine evaluation: The researchers randomly selected 100 chain steps for each tool steps) to build a machine evaluation test platform, with an average of 27 final steps and 73 intermediate tool call steps, where the final step is evaluated using the Rouge-L metric, and the intermediate step is evaluated using the ExactMatch metric.

#Manual evaluation: in weather, map, Randomly select 10 queries from the stock, translation, chemistry and WolframAlpha tools, and then evaluate the pass rate of the tool calling process, the final answer, and the comparison with the final answer of ChatGPT.

ChatGPT Evaluation: Automatic evaluation of LLaMA and ChatGPT answer and tool usage chains via ChatGPT.

The evaluation results are as follows (the higher the score, the better). It can be seen that ToolLLaMA has the same or better performance than ChatGPT in different scenarios.

Tool Learning

At Tsinghua University In a paper jointly published by well-known colleges and universities at home and abroad, such as Renmin University of China, Renmin University of Posts and Telecommunications, and Beijing University of Posts and Telecommunications, a systematic study of tool learning was conducted, and the background of tool learning was introduced, including the origin of cognition and the paradigm shift of the basic model. and the complementary role of tools and models.

Paper link: https://arxiv.org/pdf/2304.08354.pdf

The article also reviews existing tool learning research, including tool-enhanced and tool-oriented learning, and formulates a general tool learning framework: starting from understanding user instructions, the model should learn to decompose a complex task into Several subtasks, dynamically adapt the plan through reasoning, and conquer each subtask efficiently by choosing the right tool.

The article also discusses how to train models to improve tool usage and promote the popularization of tool learning.

Considering the lack of systematic tool learning evaluation in previous work, the researchers conducted experiments with 17 representative tools and demonstrated the performance of the current base model in skillfully utilizing the tools. potential.

The paper ends by discussing several open issues in tool learning that require further research, such as ensuring safe and trustworthy tool use, implementing tool creation with basic models, and solving personalization difficult problem.

Reference materials:

https://github.com/OpenBMB/ToolBench

The above is the detailed content of Tsinghua University and other open source "tool learning benchmark" ToolBench, fine-tuning model ToolLLaMA performance surpasses ChatGPT. For more information, please follow other related articles on the PHP Chinese website!