Sweep 99 sub-missions with MoE! Zhejiang University and others proposed a new general robot strategy GeRM

Multi-task robot learning is of great significance in dealing with diverse and complex scenarios. However, current methods are limited by performance issues and difficulties in collecting training datasets.

This paper proposes GeRM (General Robot Model), where researchers use offline reinforcement learning to optimize data utilization strategies, learning from demonstrations and sub-optimal data, thereby surpassing human demonstrations limitations.

Authors: Song Wenxuan, Zhao Han, Ding Pengxiang, Cui Can, Lu Shangke, Fan Yaning, Wang Donglin

Unit: West Lake University, Zhejiang University

Paper address: https://arxiv.org/abs/2403.13358

Project address: https://songwxuan.github.io/GeRM/

Then a Transformer-based vision-language-action model is used to process multi-modal input and output actions.

By introducing an expert hybrid structure, GeRM achieves faster inference speed and higher overall model capacity, thus solving the problem of limited reinforcement learning parameters and improving multi-task performance. Model performance during learning while controlling computational cost.

Through a series of experiments, it is proven that GeRM outperforms other methods in all tasks, while verifying its efficiency in the training and inference processes.

In addition, the researchers also provided the QUARD-Auto data set to support training. The construction of this data set follows the new paradigm of data automation collection proposed in the article. This method can reduce the number of collection robots. The cost of data drives progress in the multi-task learning community.

Main contributions:

#1. Proposed a hybrid expert model for four-legged reinforcement learning for the first time. Train on mixed-quality data with the potential to learn optimal policies.

2. Compared with existing methods, GeRM shows a higher success rate when only activating 1/2 of its own parameters, activating the emergence ability, and at the same time during the training process A better data utilization strategy is demonstrated in .

3. Proposed a paradigm for fully automatic robot data set collection, and collected a large-scale open source data set.

Method

The GeRM network structure is shown in Figure 1. The visual-linguistic input including demonstration data and failure data is input to 8 after passing through the encoder and tokenizer respectively. The decoder uses a layer of mixed expert structure to generate action tokens, which are eventually converted into discrete robot action data and deployed to the robot through the underlying strategy. In addition, we use reinforcement learning for training.

Figure 1 GeRM network structure diagram

GeRM Decoder is an architecture model including Transformer Decoder, in which A feedforward network (FFN) was selected from a set of 8 different expert networks.

At each layer, for each token, the gating network selects two experts to process the token and combine their outputs in a weighted manner.

Different experts are good at different tasks/different action dimensions to solve problems in different scenarios, thereby learning a common model across multiple tasks. This architecture expands the amount of network parameters while keeping the computational cost essentially unchanged.

Figure 2 Decoder structure diagram

We propose an automatic paradigm to collect robot multi-mode status data. In this way, we constructed QUARD-Auto, a large-scale robotics dataset containing a combination of demonstration and suboptimal data. It includes 5 tasks and 99 subtasks, with a total of 257k trajectories. We will open source to promote the development of the robotics community.

Table 1 Introduction to the data set

Figure 3 Data Volume statistics

Experiments

#We conducted a comprehensive and robust series of experiments covering all 99 subtasks, each of which was carefully tested on 400 trajectories.

As shown in Table 1, GeRM has the highest success rate among all tasks. Compared with RT-1 and other variants of GeRM, it effectively learns from mixed-quality data, outperforms other methods, and exhibits superior capabilities in multiple tasks. At the same time, the MoE module balances computational cost and performance by activating some parameters during inference.

Table 2 Multi-task comparison experiment

GeRM shows commendable training efficiency. Compared with other methods, GeRM achieves extremely low loss and high success rate with only a few batches, highlighting GeRM's ability to optimize data utilization strategies.

Figure 4 Success rate/Loss change curve

GeRM demonstrates dynamic adaptive path planning emergent ability. As shown in the video, the quadruped robot has a limited field of view in the initial position, making it difficult to determine the direction of movement. To avoid the obstacle, it randomly chooses to turn left.

Subsequently, after encountering erroneous visual input, the robot performed a substantial reorientation to align with the correct target outside the original field of view. It then continues toward its destination, ultimately completing its mission.

It is worth noting that such trajectories do not belong to the distribution of our training data set. This demonstrates GeRM's emergent capabilities for dynamic adaptive path planning in the context of a scene, i.e., its ability to make decisions based on visual perception, plan future paths, and change next steps as needed.

Figure 5 Emergent Capability

The above is the detailed content of Sweep 99 sub-missions with MoE! Zhejiang University and others proposed a new general robot strategy GeRM. For more information, please follow other related articles on the PHP Chinese website!