Tsinghua University and Ideal proposed DriveVLM, a visual large language model to improve autonomous driving capabilities

In the field of autonomous driving, researchers are also exploring the direction of large models such as GPT/Sora.

Compared with generative AI, autonomous driving is also one of the most active research and development areas in recent AI. A major challenge in building a fully autonomous driving system is AI's scene understanding, which involves complex, unpredictable scenarios such as severe weather, complex road layouts, and unpredictable human behavior.

The current autonomous driving system usually consists of three parts: 3D perception, motion prediction and planning. Specifically, 3D perception is mainly used to detect and track familiar objects, but its ability to identify rare objects and their attributes is limited; while motion prediction and planning mainly focus on the trajectory actions of objects, but usually ignore the relationship between objects and vehicles. decision-level interactions between These limitations may affect the accuracy and safety of autonomous driving systems when handling complex traffic scenarios. Therefore, future autonomous driving technology needs to be further improved to better identify and predict various types of objects, and to plan the vehicle's driving path more effectively to improve the intelligence and reliability of the system

implementation The key to autonomous driving is to transform a data-driven approach into a knowledge-driven approach, which requires training large models with logical reasoning capabilities. Only in this way can the autonomous driving system truly solve the long tail problem and move towards L4 capabilities. Currently, as large models like GPT4 and Sora continue to emerge, the scale effect has also demonstrated powerful few-shot/zero-shot capabilities, which has led people to consider a new development direction.

The latest research paper comes from the Cross Information Institute of Tsinghua University and Li Auto, in which they introduce a new model called DriveVLM. This model is inspired by the visual language model (VLM) emerging in the field of generative artificial intelligence. DriveVLM has demonstrated excellent capabilities in visual understanding and reasoning.

This work is the first in the industry to propose an autonomous driving speed control system. Its method fully combines the mainstream autonomous driving process with a large-scale model process with logical thinking capabilities, and is the first time to successfully deploy a large-scale model to Terminal for testing (based on Orin platform).

DriveVLM covers a Chain-of-Though (CoT) process, including three main modules: scenario description, scenario analysis and hierarchical planning. In the scene description module, language is used to describe the driving environment and identify key objects in the scene; the scene analysis module deeply studies the characteristics of these key objects and their impact on autonomous vehicles; while the hierarchical planning module gradually formulates plans from the elements Actions and decisions are described to waypoints.

These modules correspond to the perception, prediction, and planning steps of traditional autonomous driving systems, but the difference is that they handle object perception, intent-level prediction, and task-level planning, which have been very challenging in the past.

Although VLMs perform well in visual understanding, they have limitations in spatial basis and reasoning, and their computing power requirements pose challenges to the speed of end-side reasoning. Therefore, the authors further propose DriveVLMDual, a hybrid system that combines the advantages of DriveVLM and traditional systems. DriveVLM-Dual optionally integrates DriveVLM with traditional 3D perception and planning modules such as 3D object detectors, occupancy networks, and motion planners, enabling the system to achieve 3D grounding and high-frequency planning capabilities. This dual-system design is similar to the slow and fast thinking processes of the human brain and can effectively adapt to different complexities in driving scenarios.

The new research also further clarifies the definition of scene understanding and planning (SUP) tasks and proposes some new evaluation metrics to evaluate the capabilities of DriveVLM and DriveVLM-Dual in scene analysis and meta-action planning. In addition, the authors performed extensive data mining and annotation work to build an in-house SUP-AD dataset for the SUP task.

After extensive experiments on the nuScenes dataset and our own dataset, the superiority of DriveVLM was demonstrated, especially with a small number of shots. Furthermore, DriveVLM-Dual surpasses state-of-the-art end-to-end motion planning methods.

Paper "DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models"

##Paper link: https://arxiv.org/abs/ 2402.12289

Project connection: https://tsinghua-mars-lab.github.io/DriveVLM/

The overall process of DriveVLM is shown in Figure 1:

1. Encode continuous frame visual images and interact with LMM through the feature alignment module;

2. Start from the scene description to guide the thinking of the VLM model, first guide the time, scene, Static scenes such as lane environments are used to guide key obstacles that affect driving decisions;

3. analyzes key obstacles and matches them through traditional 3D detection and VLM-understood obstacles to further improve Confirm the effectiveness of obstacles and eliminate illusions, describe the characteristics of key obstacles in this scenario and their impact on our driving;

4. Gives key "meta-decisions", such as deceleration, parking, turning left and right, etc., and then gives a description of the driving strategy based on the meta-decisions, and finally gives the future driving trajectory of the host vehicle.

^{Figure 1. DriveVLM and DriveVLM-Dual model pipeline. A sequence of images is processed by a large visual language model (VLM) to perform special chain-of-thought (CoT) reasoning to derive driving planning results. Large VLM involves a visual transformer encoder and a large language model (LLM). A visual encoder produces image tags; an attention-based extractor then aligns these tags with an LLM; and finally, the LLM performs CoT inference. The CoT process can be divided into three modules: scenario description, scenario analysis, and hierarchical planning.}

DriveVLM-Dual is a hybrid system that improves the decision-making and planning capabilities of traditional autonomous driving pipelines by leveraging DriveVLM’s comprehensive understanding of the environment and recommendations for decision trajectories. It incorporates 3D perception results into verbal cues to enhance 3D scene understanding and further refines trajectory waypoints with a real-time motion planner.

Although VLMs are good at identifying long-tail objects and understanding complex scenes, they often struggle to accurately understand the spatial location and detailed motion status of objects, a shortcoming that poses a significant challenge. To make matters worse, the huge model size of VLM results in high latency, hindering the real-time response capability of autonomous driving. To address these challenges, the author proposes DriveVLM-Dual, which allows DriveVLM and traditional autonomous driving systems to cooperate. This new approach involves two key strategies: key object analysis combined with 3D perception to give high-dimensional driving decision information, and high-frequency trajectory refinement.

In addition, to fully realize the potential of DriveVLM and DriveVLMDual in handling complex and long-tail driving scenarios, the researchers formally defined a task called scene understanding planning, as well as a set of evaluation metrics. Furthermore, the authors propose a data mining and annotation protocol to manage scene understanding and planning datasets.

In order to fully train the model, the author has newly developed a set of Drive LLM annotation tools and annotation solutions, which are combined with multiple methods such as automated mining, perceptual algorithm pre-brushing, GPT-4 large model summary and manual annotation. , forming the current set of efficient annotation solutions. Each Clip data contains dozens of annotation contents.

## 图 2 2. Annotation sample of the SUP-AD dataset.

The authors also propose a comprehensive data mining and annotation pipeline, as shown in Figure 3, to build Scene Understanding for Planning in SUP-AD (Scene Understanding for Planning in Autonomous Driving) dataset, containing 100k images and 1000k image-text pairs. Specifically, the authors first perform long-tail object mining and challenging scene mining from a large database to collect samples, and then select a keyframe from each sample and further perform scene annotation.

Figure 3. Data mining and annotation pipeline for building scenario understanding and planning datasets (above). Examples of scenarios randomly sampled from the dataset (below) demonstrate the diversity and complexity of the dataset.

SUP-AD is divided into training, validation and testing parts with a ratio of 7.5:1:1.5. The authors train the model on the training split and use newly proposed scene description and meta-action metrics to evaluate the model performance on the validation/test split.

nuScenes dataset is a large-scale urban scene driving dataset with 1000 scenes, each lasting about 20 seconds. Keyframes are annotated uniformly at 2Hz across the entire dataset. Here, the authors adopt displacement error (DE) and collision rate (CR) as indicators to evaluate the model's performance on verification segmentation.

###The authors demonstrate the performance of DriveVLM with several large-scale visual language models and compare them with GPT-4V, as shown in Table 1. DriveVLM utilizes Qwen-VL as its backbone, which achieves the best performance compared to other open source VLMs and is characterized by responsiveness and flexible interaction. The first two large models have been open sourced and used the same data for fine-tuning training. GPT-4V uses complex prompts for prompt engineering. ###

^{Table 1. Test set results on the SUP-AD data set. The official API of GPT-4V is used here, and for Lynx and CogVLM, training splits are used for fine-tuning.}

As shown in Table 2, DriveVLM-Dual achieves state-of-the-art performance on nuScenes planning tasks when paired with VAD. This shows that the new method, although tailored for understanding complex scenes, also performs well in ordinary scenes. Note that DriveVLM-Dual improves significantly over UniAD: the average planning displacement error is reduced by 0.64 meters and the collision rate is reduced by 51%.

^{Table 2. Planning results for nuScenes validation dataset. DriveVLM-Dual achieves optimal performance. †Represents perception and occupancy prediction results using Uni-AD. ‡ Indicates working with VAD, where all models take ego states as input. Figure 4. Qualitative results of DriveVLM. The orange curve represents the model's planned future trajectory over the next 3 seconds.}

Qualitative results of DriveVLM are shown in Figure 4. In Figure 4a, DriveVLM accurately predicts current scene conditions combined with thoughtful planning decisions about cyclists approaching us. DriveVLM also effectively understands the hand signals of the traffic police ahead to signal the self-vehicle to proceed, and also takes into account the person riding a tricycle on the right to make correct driving decisions. These qualitative results demonstrate the DriveVLM model’s superior ability to understand complex scenarios and develop appropriate driving plans.

## 图 7: Various driving scenarios in SUP-AD data concentration.

# 图 9. Sup-AD data concentration cow cluster and herds. A herd of cattle is moving slowly in front of the car, requiring the policy to reason that the car is moving slowly and keeping a safe distance from the cattle. Figure 16. Visualization of DriveVLM output. DriveVLM can accurately detect fallen trees and their locations, then plan an appropriate detour.

The above is the detailed content of Tsinghua University and Ideal proposed DriveVLM, a visual large language model to improve autonomous driving capabilities. For more information, please follow other related articles on the PHP Chinese website!