Generative models build interactive real-world simulators, which LeCun thinks is pretty cool

Generative models trained on Internet data revolutionize the way text, image and video content is created. Some researchers predict that perhaps the next milestone in generative models will be the ability to simulate all aspects of human experience in the world, such as how to drive a car on the road or how to prepare meals.

Nowadays, with the help of very comprehensive real-world simulators, humans can interact with different scenes and objects, and robots can also learn from simulated experiences. This avoids the risk of physical damage.

However, one of the major obstacles to building such a real-world simulator lies in the available data sets. Although there are billions of text, images, and video clips on the Internet, different data sets cover different axes of information, and these data sets must be brought together to simulate a realistic experience of the world. For example, paired text image data contains rich scenes and objects but few actions; video subtitles and question and answer data contain rich high-level activity descriptions but few low-level motion details; human activity data contains rich human actions, But there are few mechanical movements; while robot data contains rich robot movements, but the number is limited

The information differences listed above are natural and difficult to overcome, which makes it difficult to build a system designed to capture Real-world simulators bring difficulties with real-world experiences.

In this article, researchers from UC Berkeley, Google DeepMind, MIT and other institutions explore UniSim, a universal simulator that learns real-world interactions through generative models, taking a step toward building a universal simulator. the first step. For example, UniSim can simulate how humans and agents interact with the world by simulating high-level instructions such as "open a drawer" and the visual results of low-level instructions.

• Paper address: https://arxiv.org/pdf/2310.06114.pdf
• Paper home page: https://universal-simulator.github.io/unisim/

This paper combines large amounts of data (including Internet text-image pairs, rich data from navigation, human activities, robot actions, etc., and data from simulation and rendering) into a conditional video generation framework. Then by carefully orchestrating rich data along different axes, this paper shows that UniSim can successfully merge experience from different axes of data and generalize beyond the data to enable rich interactions through fine-grained motion control of static scenes and objects.

The following video demonstrates how UniSim simulates an example with a long interaction horizon. The video shows that UniSim simulates eight robot action instructions in one go:

UniSim’s simulation of human actions:

UniSim’s simulation deployment of RL strategy is as follows:

Meta Chief AI Scientist Yann LeCun and NVIDIA Senior Research Scientist Jim Fan and other industry experts forwarded this research. LeCun gave this a "cool" evaluation

Jim Fan said that this work is very interesting. The video diffusion model is used as a data-driven physics simulation in which an agent can plan, explore, and learn optimal actions without touching the robotic hardware or causing any damage. It can be said that LLM is not only an operating system, but also a complete reality simulator

The first author of the paper is a Ph.D. from the University of California, Berkeley Student Sherry Yang said, “Learning real-world models is becoming a reality.”

Simulating real-world interactions

As shown in Figure 3, UniSim can simulate a series of rich actions in the kitchen scene, including washing hands, holding bowls, cutting carrots and Dry your hands. The top right of Figure 3 shows different switches, while the bottom of Figure 3 shows two navigation scenarios

The content that needs to be rewritten is: Correspondence The navigation scene

in the lower right corner of Figure 3 corresponds to the navigation scene

in the lower right corner of Figure 3 above. Figure 4 below shows an example of UniSim autoregressively simulating 8 interactions sequentially. In terms of long-term simulation

UniSim not only supports rich actions and Long-range interaction can also achieve highly diverse and random environmental changes. For example, after removing the top towel, the objects displayed have diversity (see Figure 5 below, left)

UniSim Real World Migration result. The real value of UniSim is in simulating the real world, and Figure 7 shows the language plan generated by VLM, the video generated by UniSim based on the language plan, and the execution on a real robot.

In addition to testing UniSim’s migration capabilities in the real world, this article also conducted a simulator-based evaluation. The results are shown in Table 2:

Real-world simulator for reinforcement learning

The experiment also evaluates UniSim's ability to simulate real-world How well does the robot perform various actions? The robot moves the endpoint left, right, down, and up by repeatedly performing low-level control operations for about 20-30 steps. Table 3 shows that RL training significantly improves the performance of the VLA policy in various tasks, especially in tasks such as pointing to the blue block. We then directly deploy the zero-shot RL policy trained in UniSim onto a real robot, as shown in Figure 8 (bottom row).

The above is the detailed content of Generative models build interactive real-world simulators, which LeCun thinks is pretty cool. For more information, please follow other related articles on the PHP Chinese website!