Control more than 100,000 AI models with one click, HuggingFace creates an 'APP Store” for ChatGPT-like models

From chatting to programming to supporting various plug-ins, the powerful ChatGPT has long been no longer a simple conversation assistant, but has been moving towards the "management" of the AI ​​world.

On March 23, OpenAI announced that ChatGPT began to support various third-party plug-ins, such as the famous science and engineering artifact Wolfram Alpha. With the help of this artifact, ChatGPT, who was originally a chicken and a rabbit in the same cage, became a top student in science and engineering. Many people on Twitter commented that the launch of the ChatGPT plug-in looks a bit like the launch of the iPhone App Store in 2008. This also means that AI chatbots are entering a new stage of evolution - the "meta app" stage.

## Then, in early April, researchers from Zhejiang University and Microsoft Asia Research proposed a method called "HuggingGPT" ” important method can be regarded as a large-scale demonstration of the above route. HuggingGPT allows ChatGPT to act as a controller (can be understood as a management layer), which manages a large number of other AI models to solve some complex AI tasks. Specifically, HuggingGPT uses ChatGPT for task planning when it receives a user request, selects a model based on the feature description available in HuggingFace, executes each subtask with the selected AI model, and aggregates the response based on the execution results.

This approach can make up for many shortcomings of current large models, such as limited modalities that can be processed, and in some aspects it is not as good as professional models.

Although the HuggingFace model is scheduled, HuggingGPT is not an official product of HuggingFace after all. Just now, HuggingFace finally took action.

Similar to HuggingGPT, they have launched a new API - HuggingFace Transformers Agents. With Transformers Agents, you can control more than 100,000 Hugging Face models to complete various multi-modal tasks.

For example, in the example below, you want Transformers Agents to explain out loud what is depicted on the picture. It will try to understand your instructions (Read out loud the content of the image), then convert them into prompts, and select the appropriate models and tools to complete the tasks you specify.

NVIDIA AI scientist Jim Fan commented: This day has finally come. This is an important step towards “Everything APP”. .

However, some people say that this is not the same as AutoGPT’s automatic iteration. It is more like eliminating the need to write prompt and manually specifying these steps for tools, it is too early for the Master of All Things APP.

##Transformers Agents Address: https://huggingface.co/docs/transformers/transformers_agents Transformers Agents How to use?

At the same time as the release, HuggingFace released the Colab address, and anyone can try it out:

https://huggingface. co/docs/transformers/en/transformers_agents

In short, it provides a natural language API on top of transformers: first define a set of curated tools, and An agent was designed to interpret natural language and use these tools.

Furthermore, Transformers Agents are extensible by design.

The team has identified a set of tools that can be empowered to the agent. Here is the list of integrated tools:

• Documentation Q&A: Given an image Format document (such as PDF), answer questions about the document (Donut)
• Text Q&A: Given a long text and a question, answer the question in the text (Flan-T5 )
• Unconditional Image Caption: Add a caption to an image (BLIP)
• Image Q&A: Given an image, answer questions about this Image problem (VILT)
• Image segmentation: Given an image and a prompt, output the segmentation mask (CLIPSeg) of the prompt
• Voice Text to Text: Given a recording of a person speaking, transcribe the speech into text (Whisper)
• Text to Speech: Convert text to speech (SpeechT5)
• Zero-shot text classification: Given a text and a list of tags, determine which tag the text most corresponds to (BART)
• Text summary: summarize it in one or a few sentences A long text (BART)
• Translation: Translate text into a given language (NLLB)

These tools are integrated in transformers, or can be used manually:

<code>from transformers import load_tooltool = load_tool("text-to-speech")audio = tool("This is a text to speech tool")</code>

Users can also push the tool's code to Hugging Face Space or the model repository to utilize the tool directly through the agent, such as:

• Text Downloader: Download text from web URL
• Text to image: Generate image according to prompt, using Stable Diffusion
• Image conversion: Modify the image given the initial image and prompt, using instruct pix2pix stable diffusion
• Text to video: generated according to prompt Small video, using damo-vilab

##For specific gameplay, let’s first look at a few examples of HuggingFace:

Generate image description :

<code>agent.run("Caption the following image", image=image)</code>

Read text:

<code>agent.run("Read the following text out loud", text=text)</code>

Input: A beaver is swimming in the water

Output:

tts_exampleAudio:00:0000:01

##Read File:

Before running agent.run, you need to instantiate a large language model agent. It supports OpenAI models and open source models such as BigCode and OpenAssistant.

First, please install the agents add-on to install all default dependencies:

<code>pip install transformers[agents]</code>

To use the openAI model, you need to install the dependencies after Instantiate an "OpenAiAgent" openai:

<code>pip install openaifrom transformers import OpenAiAgentagent = OpenAiAgent(model="text-davinci-003", api_key="<your_api_key>")</code>

To use BigCode or OpenAssistant, first log in to access the inference API:

<code>from huggingface_hub import loginlogin("<YOUR_TOKEN>")</code>

Then, instance Agent:

<code>from transformers import HfAgentStarcoderagent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoder")StarcoderBaseagent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoderbase")OpenAssistantagent = HfAgent(url_endpoint="https://api-inference.huggingface.co/models/OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5")</code>

If the user has his own inference endpoint for this model (or another model), he can replace the URL above with his own URL endpoint.

接下来，我们了解一下 Transformers Agents 提供的两个 API：

单次执行

单次执行是在使用智能体的 run () 方法时：

<code>agent.run("Draw me a picture of rivers and lakes.")</code>

它会自动选择适合要执行的任务的工具并适当地执行，可在同一指令中执行一项或多项任务（不过指令越复杂，智能体失败的可能性就越大）。

<code>agent.run("Draw me a picture of the sea then transform the picture to add an island")</code>

每个 run () 操作都是独立的，因此可以针对不同的任务连续运行多次。如果想在执行过程中保持状态或将非文本对象传递给智能体，用户可以通过指定希望智能体使用的变量来实现。例如，用户可以生成第一张河流和湖泊图像，并通过执行以下操作要求模型更新该图片以添加一个岛屿：

<code>picture = agent.run("Generate a picture of rivers and lakes.")updated_picture = agent.run("Transform the image in picture to add an island to it.", picture=picture)</code>

当模型无法理解用户的请求并混合使用工具时，这会很有帮助。一个例子是：

<code>agent.run("Draw me the picture of a capybara swimming in the sea")</code>

在这里，模型可以用两种方式解释：

• 让 text-to-image 水豚在海里游泳
• 或者，生成 text-to-image 水豚，然后使用 image-transformation 工具让它在海里游泳

如果用户想强制执行第一种情况，可以通过将 prompt 作为参数传递给它来实现：

<code>agent.run("Draw me a picture of the prompt", prompt="a capybara swimming in the sea")</code>

基于聊天的执行 

智能体还有一种基于聊天的方法：

<code>agent.chat("Generate a picture of rivers and lakes")</code>

<code>agent.chat ("Transform the picture so that there is a rock in there")</code>

这是一种可以跨指令保持状态时。它更适合实验，但在单个指令上表现更好，而 run () 方法更擅长处理复杂指令。如果用户想传递非文本类型或特定 prompt，该方法也可以接受参数。

The above is the detailed content of Control more than 100,000 AI models with one click, HuggingFace creates an 'APP Store” for ChatGPT-like models. For more information, please follow other related articles on the PHP Chinese website!