Reinforcement learning is on the cover of Nature again, and the new paradigm of autonomous driving safety verification significantly reduces test mileage

Introduce dense reinforcement learning and use AI to verify AI.

Rapid advances in autonomous vehicle (AV) technology have us on the cusp of a transportation revolution on a scale not seen since the advent of the automobile a century ago. Autonomous driving technology has the potential to significantly improve traffic safety, mobility, and sustainability, and therefore has attracted the attention of industry, government agencies, professional organizations, and academic institutions.

The development of autonomous vehicles has come a long way over the past 20 years, especially with the advent of deep learning. By 2015, companies were starting to announce that they would be mass-producing AVs by 2020. But so far, no level 4 AV is available on the market.

There are many reasons for this phenomenon, but the most important is that the safety performance of self-driving cars is still significantly lower than that of human drivers. For the average driver in the United States, the probability of a collision in the natural driving environment (NDE) is approximately 1.9 × 10^−6 per mile. By comparison, the disengagement rate for state-of-the-art autonomous vehicles is about 2.0 × 10^−5/mile, according to California’s 2021 Disengagement Reports.

Note: The disengagement rate is an important indicator for evaluating the reliability of autonomous driving. It describes the number of times the system requires the driver to take over every 1,000 miles of operation. The lower the disengagement rate of the system, the better the reliability. When the disengagement rate is equal to 0, it means that the autonomous driving system has reached the driverless level to some extent.

Although the disengagement rate can be criticized for being biased, it has been widely used to evaluate the safety performance of autonomous vehicles.

A key bottleneck in improving the safety performance of autonomous vehicles is the low efficiency of safety verification. It is currently popular to test the non-destructive testing of autonomous vehicles through a combination of software simulation, closed test track and road testing. As a result, AV developers must incur significant economic and time costs for evaluation, hindering the progress of AV deployment.

Verifying AV security performance in an NDE environment is very complex. For example, driving environments are complex in space and time, so the variables required to define such environments are high-dimensional. As the dimensionality of variables increases exponentially, so does the computational complexity. In this case, deep learning models are difficult to learn even given large amounts of data.

In this article, researchers from the University of Michigan, Ann Arbor, Tsinghua University and other institutions propose a dense deep-reinforcement learning (D2RL) method to solve this challenge.

The study appears on the cover of Nature.

• Paper address: https://www.nature.com/articles/s41586-023-05732-2
• Project address: https ://github.com/michigan-traffic-lab/Dense-Deep-Reinforcement-Learning

The thesis was awarded a master's degree, and he is currently a Tenure-Track Assistant Professor in the Department of Automation, Tsinghua University. , Additionally, he is an Assistant Research Scientist at the University of Michigan Transportation Research Institute (UMTRI). He received his bachelor's and doctoral degrees from the Department of Automation, Tsinghua University, in 2014 and 2019, under the supervision of Professor Zhang Yi. From 2017 to 2019, he was a visiting doctoral student in Civil and Environmental Engineering at the University of Michigan, studying under Professor Henry X. Liu (corresponding author of this article).

Research Introduction

The basic idea of ​​the D2RL method is to identify and remove non-safety-critical data, and use safety-critical data to train the neural network. Since only a small portion of the data is security-critical, the remaining data will be heavily densified with information.

Compared with the DRL method, the D2RL method can significantly reduce the variance of the policy gradient estimate by multiple orders of magnitude without losing unbiasedness. This significant variance reduction can enable neural networks to learn and complete tasks that are intractable for DRL methods.

For AV testing, this research utilizes the D2RL method to train background vehicles (BV) through neural networks to learn when to perform what adversarial operations, aiming to improve testing efficiency. D2RL can reduce the test mileage required for AVs by multiple orders of magnitude in an AI-based adversarial testing environment while ensuring unbiased testing.

The D2RL method can be applied to complex driving environments, including multiple highways, intersections, and roundabouts, which was not possible with previous scenario-based methods. Moreover, the method proposed in this study can create intelligent testing environments that use AI to verify AI. This is a paradigm shift that opens the door for accelerated testing and training of other safety-critical systems.

In order to prove that the AI-based testing method is effective, this study trained BV using a large-scale actual driving data set, and conducted simulation experiments and field experiments on physical test tracks. The experimental results are as follows Figure 1 shown.

Dense Deep Reinforcement Learning

To take advantage of AI technology, this study formulated the AV testing problem as a Markov Decision Process (MDP), The operation of BV is determined based on the current status information. The study aims to train a policy (DRL agent) modeled by a neural network that controls the actions of BVs interacting with AVs to maximize evaluation efficiency and ensure unbiasedness. However, as mentioned above, due to the limitations of dimensionality and computational complexity, it is difficult or even impossible to learn effective policies if the DRL method is directly applied.

Since most states are non-critical and cannot provide information for security-critical events, D2RL focuses on removing data from these non-critical states. For AV testing problems, many security metrics can be leveraged to identify critical states with varying efficiency and effectiveness. The criticality metric utilized in this study is an external approximation of the AV collision rate within a specific time frame of the current state (e.g., 1 second). The study then edited the Markov process, discarding data for non-critical states, and used the remaining data for policy gradient estimation and Bootstrap for DRL training.

As shown in Figure 2 below, compared to DRL, the advantage of D2RL is that it can maximize the reward during the training process.

AV Simulation Test

To evaluate the accuracy, efficiency, scalability and generality of the D2RL method, this study was conducted simulation test. For each test set, the study simulated a fixed distance of traffic travel and then recorded and analyzed the test results, as shown in Figure 3 below.

In order to further study the scalability and generalization of D2RL, this study conducted AV-I models with different number of lanes (2 lanes and 3 lanes) and driving distance. (400 m, 2 km, 4 km and 25 km) experiments. This article examines 25-kilometer trips because the average commuter in the United States travels approximately 25 kilometers one-way. The results are shown in Table 1:

The above is the detailed content of Reinforcement learning is on the cover of Nature again, and the new paradigm of autonomous driving safety verification significantly reduces test mileage. For more information, please follow other related articles on the PHP Chinese website!