Further accelerating implementation: compressing the end-to-end motion planning model of autonomous driving

Original title: On the Road to Portability: Compressing End-to-End Motion Planner for Autonomous Driving

Paper link: https://arxiv.org/pdf/2403.01238.pdf

Code link: https://github.com/tulerfeng/PlanKD

Author affiliation: Beijing Institute of Technology ALLRIDE.AI Hebei Provincial Key Laboratory of Big Data Science and Intelligent Technology

Thesis Idea

The end-to-end motion planning model is equipped with a deep neural network and has shown great potential in realizing fully autonomous driving. However, overly large neural networks make them unsuitable for deployment on resource-constrained systems, which undoubtedly require more computing time and resources. To address this problem, knowledge distillation offers a promising approach by compressing models by having a smaller student model learn from a larger teacher model. Nonetheless, how to apply knowledge distillation to compress motion planners has so far been unexplored. This paper proposes PlanKD, the first knowledge distillation framework tailored for compressed end-to-end motion planners. First, given that driving scenarios are inherently complex and often contain information that is irrelevant to planning or even noisy, transferring this information would not be beneficial to the student planner. Therefore, this paper designs a strategy based on information bottleneck, which only distills planning-related information instead of migrating all information indiscriminately. Second, different waypoints in the output planned trajectory may vary in importance to motion planning, and slight deviations in some critical waypoints may lead to collisions. Therefore, this paper designs a safety-aware waypoint-attentive distillation module to assign adaptive weights to different waypoints based on importance to encourage student models to imitate more critical waypoints more accurately, thereby improving overall safety. Experiments show that our PlanKD can significantly improve the performance of small planners and significantly reduce their reference time.

Main Contributions:

1. This paper constructs the first attempt to explore dedicated knowledge distillation methods to compress end-to-end motion planners in autonomous driving.
2. This paper proposes a general and innovative framework PlanKD, which enables student planners to inherit planning-related knowledge in the middle layer and facilitates accurate matching of key waypoints to improve safety. .
3. Experiments show that PlanKD in this article can significantly improve the performance of small planners, thereby providing a more portable and efficient solution for deployment with limited resources.

Network Design:

End-to-end motion planning has recently emerged as a promising direction in autonomous driving [3, 10, 30, 31, 40, 47, 48], which directly maps raw sensor data to planned actions. This learning-based paradigm exhibits the advantages of reducing heavy reliance on hand-crafted rules and mitigating error accumulation within complex cascade modules (usually detection-tracking-prediction-planning) [40, 48]. Despite their success, the bulky architecture of deep neural networks in motion planners poses challenges for deployment in resource-constrained environments, such as autonomous delivery robots that rely on the computing power of edge devices. Furthermore, even in conventional vehicles, computing resources on-board devices are often limited [34]. Therefore, directly deploying deep and large planners inevitably requires more computing time and resources, which makes it challenging to respond quickly to potential hazards. To alleviate this problem, a straightforward approach is to reduce the number of network parameters by using a smaller backbone network, but this paper observes that the performance of the end-to-end planning model will drop sharply, as shown in Figure 1. For example, although the inference time of InterFuser [33], a typical end-to-end motion planner, was reduced from 52.9M to 26.3M, its driving score also dropped from 53.44 to 36.55. Therefore, it is necessary to develop a model compression method suitable for end-to-end motion planning.

In order to obtain a portable motion planner, this article uses knowledge distillation [19] to compress the end-to-end motion planning model. Knowledge distillation (KD) has been widely studied for model compression in various tasks, such as object detection [6, 24], semantic segmentation [18, 28], etc. The basic idea of ​​these works is to train a simplified student model by inheriting knowledge from a larger teacher model and use the student model to replace the teacher model during deployment. While these studies have achieved significant success, directly applying them to end-to-end motion planning leads to suboptimal results. This stems from two emerging challenges inherent in motion planning tasks: (i) Driving scenarios are complex in nature [46], involving multiple dynamic and static objects, complex background scenes, and multifaceted roads and traffic Diverse information including information. However, not all of this information is useful for planning. For example, background buildings and distant vehicles are irrelevant or even noisy to planning [41], while nearby vehicles and traffic lights have a deterministic impact. Therefore, it is crucial to automatically extract only planning-relevant information from the teacher model, which previous KD methods cannot achieve. (ii) Different waypoints in the output planning trajectory usually have different importance for motion planning. For example, when navigating an intersection, waypoints in a trajectory that are close to other vehicles may have higher importance than other waypoints. This is because at these points, the self-vehicle needs to actively interact with other vehicles, and even small deviations can lead to collisions. However, how to adaptively determine key waypoints and accurately imitate them is another significant challenge of previous KD methods.

To address the above two challenges, this paper proposes the first knowledge distillation method tailored for end-to-end motion planners in compressed autonomous driving, called PlanKD. First, this paper proposes a strategy based on the information bottleneck principle [2], whose goal is to extract planning-related features that contain minimal and sufficient planning information. Specifically, this paper maximizes the mutual information between the extracted planning-related features and the true value of the planning state defined in this paper, while minimizing the mutual information between the extracted features and intermediate feature maps. This strategy enables this paper to extract key planning-relevant information only at the middle layer, thereby enhancing the effectiveness of the student model. Second, in order to dynamically identify key waypoints and imitate them faithfully, this paper adopts an attention mechanism [38] to calculate each waypoint and its attention weight between it and the associated context in the bird's-eye view (BEV). To promote accurate imitation of safety-critical waypoints during distillation, we design a safety-aware ranking loss that encourages giving higher attention weight to waypoints close to moving obstacles. Accordingly, the security of student planners can be significantly enhanced. The evidence shown in Figure 1 shows that the driving score of student planners can be significantly improved with our PlanKD. Furthermore, our method can reduce the reference time by about 50% while maintaining comparable performance to the teacher planner on the Town05 Long Benchmark.

Figure 1. Schematic diagram of the performance degradation of InterFuser[33] as the number of parameters decreases on Town05 Long Benchmark [31]. By leveraging our PlanKD, we can improve the performance of compact motion planners and significantly reduce reference times. Inference times are evaluated on a GeForce RTX 3090 GPU on the server.

Figure 2. Schematic diagram of the PlanKD framework of this article. PlanKD consists of two modules: a planning-related feature distillation module that extracts planning-related features from intermediate feature maps through information bottlenecks (IB); a safety-aware waypoint-attentive distillation module that dynamically determines key waypoints and Extract knowledge from it to enhance overall security.

Experimental results:

Figure 3. Visualization of safety-aware attention weights in different driving scenarios. The green blocks represent the ego-vehicle and the yellow blocks represent other road users (e.g. cars, bicycles). The redder a waypoint is, the higher its attention weight.

Summary:

This paper proposes PlanKD, a knowledge distillation method tailored for compressed end-to-end motion planners. The proposed method can learn planning-related features through information bottlenecks to achieve effective feature distillation. Furthermore, this paper designs a safety-aware waypoint-attentive distillation mechanism to adaptively decide the importance of each waypoint for waypoint distillation. Extensive experiments validate the effectiveness of our approach, demonstrating that PlanKD can serve as a portable and secure solution for resource-limited deployments.

Citation:

Feng K, Li C, Ren D, et al. On the Road to Portability: Compressing End-to-End Motion Planner for Autonomous Driving[ J]. arXiv preprint arXiv:2403.01238, 2024.

The above is the detailed content of Further accelerating implementation: compressing the end-to-end motion planning model of autonomous driving. For more information, please follow other related articles on the PHP Chinese website!