Take a trip to the future, the first multi-view prediction + planning autonomous driving world model arrives

Recently, the concept of world model has triggered a wave of enthusiasm, but how can the field of autonomous driving watch the "fire" from afar. A team from the Institute of Automation, Chinese Academy of Sciences, proposed for the first time a new multi-view world model called Drive-WM, aiming to enhance the safety of end-to-end autonomous driving planning.

Website: https://drive-wm.github.io

Paper URL: https ://arxiv.org/abs/2311.17918

The first multi-view prediction and planning autonomous driving world model

In CVPR2023 Autonomous Driving At the seminar, the two major technology giants Tesla and Wayve showed off their black technology, and a new concept called "generative world model" became popular in the field of autonomous driving. Wayve even released the GAIA-1 generative AI model, demonstrating its stunning video scene generation capabilities. Recently, researchers from the Institute of Automation of the Chinese Academy of Sciences have also proposed a new autonomous driving world model - Drive-WM, which realizes a multi-view predictive world model for the first time and is seamlessly integrated with the current mainstream end-to-end autonomous driving planner. .

Drive-WM leverages the powerful generation capabilities of the Diffusion model to generate realistic video scenes.

Imagine that you are driving, and your on-board system is predicting future developments based on your driving habits and road conditions, and generating corresponding visual feedback to guide the selection of trajectory routes. This ability to foresee the future combined with a planner will greatly improve the safety of autonomous driving!

#Forecasting and planning based on multi-view world models.

#The combination of world model and end-to-end autonomous driving improves driving safety

## The #Drive-WM model combines the world model with end-to-end planning for the first time, opening a new chapter in the development of end-to-end autonomous driving. At each time step, the planner can use the world model to predict possible future scenarios, and then use the image reward function to fully evaluate them.

The end-to-end trajectory planning tree based on the world model can be rewritten

Adopt the best Estimation method and extended planning tree technology can achieve more effective and safer planning

##

Drive-WM explores two applications of world models in end-to-end planning through innovative research

1. Demonstrates the use of world models in the face of OOD scene robustness. Through comparative experiments, the author found that the current end-to-end planner's performance is not ideal when facing OOD situations.

The author gives the following picture. When a slight lateral offset is perturbed to the initial position, it is difficult for the current end-to-end planner to output a reasonable planned route.

The end-to-end planner has difficulty outputting a reasonable planned route when facing an OOD situation.

Drive-WM’s powerful generation capability provides new ideas for solving OOD problems. The author uses the generated videos to fine-tune the planner and learn from the OOD data, so that the planner can have better performance when facing such a scenario

2. This shows Introducing the enhanced role of future scenario evaluation in end-to-end planning

How to build a multi-view video generation model

Spatial-temporal consistency of multi-view video generation has always been a challenging problem. Drive-WM expands the capabilities of video generation by introducing temporal layer coding, and achieves multi-view video generation through view decomposition modeling. This generation method of view decomposition can greatly improve the consistency between views

Drive-WM overall model design

High-quality video generation and controllability

Drive-WM achieves high-quality multi-view video generation with excellent Controllability. It provides a variety of control options to control the generation of multi-view videos through text, scene layout, and motion information, and also provides new possibilities for future neural simulators

For example, use text to change weather and lighting:

##For example, pedestrian generation and foreground editing:

Use speed and direction control methods:

Generate rare events , such as turning around at an intersection or driving into the grass on the side

Conclusion

Drive-WM not only demonstrated its powerful multi-view video generation capabilities, but also revealed the world model and terminal There is huge potential in combining end-to-end driving models. We believe that in the future, world models can help achieve a safer, more stable, and more reliable end-to-end autonomous driving system.

The above is the detailed content of Take a trip to the future, the first multi-view prediction + planning autonomous driving world model arrives. For more information, please follow other related articles on the PHP Chinese website!