Discuss the current status and development trends of autonomous driving trajectory prediction technology

1 What is trajectory prediction

In autonomous driving, trajectory prediction is generally located at the back end of the perception module, and the front end of the control module is a connecting module. Input the state information and road structure information of the target track provided by the perception module, comprehensively consider high-precision map information, interaction information between targets, semantic information of the environment and intention information of the target, and make intentions for various perceived targets. Prediction (cut in/out, straight ahead) and trajectory prediction for a period of time in the future (ranging from 0-5s). As shown below.

ADAS system requires a certain cognitive ability of the surrounding environment information. The most basic level is to recognize the environment. The next level requires understanding the environment, and the next level requires Make predictions about the environment. After predicting the target, the controller can plan the path of the vehicle based on the predicted information, and make decisions to brake or issue warnings for possible dangerous situations. This is the significance of the trajectory prediction module.

2 Two challenges

## Trajectory prediction can be divided into short-term prediction and long-term prediction .

1. Short-term prediction is generally based on the kinematic model (CV/CA/CTRV/CTRA) to predict the trajectory for a period of time in the future based on the current target state information. Generally
2. Long-term forecasting is what the industry is currently doing. This kind of prediction is not suitable based only on the motion model. Generally, intention prediction is required and combined with some contextual information (map, interaction information between targets) to obtain good results. At this time, there are many different output forms in the industry, such as outputting the probability distribution of the trajectory, outputting multiple predicted trajectories, and outputting the most likely predicted trajectory.

There are two challenges for long-term trajectory prediction:

1. Output a possible trajectory or output all possible trajectories The trajectories are all unreasonable. If you output a predicted trajectory, you may miss the real trajectory. If you output all possible trajectories, there will be false alarms. This is unacceptable for ADAS systems. Consideration should be given to limiting the predicted trajectories to a suitable subset.
2. The more you do on trajectory prediction, the more assumptions you need to make. An extreme assumption is to assume that all objects on the road obey traffic rules. This is reasonable if used for traffic simulation functions, but is not suitable for ADAS systems, which need to be sensitive to potentially dangerous situations.

The uncertainty that affects long-term trajectory prediction mainly comes from three aspects:

1. The output of the sensing module The target state estimate is uncertain.
2. Uncertainty in driving intention prediction.
3. The uncertainty between intent recognition and vehicle mobility changes.

3 Main considerations

Four issues that should be considered for the trajectory prediction system:

1. Trajectory prediction must be sensitive to potential dangers, which is required by the meaning of trajectory prediction.
2. It is necessary to consider both the running model and the information about the intention and surrounding environment.
3. Consider the above uncertainties.
4. Consider the number of output trajectories.

4 Industry method

The following figure shows the classification method in the review paper [2] published by Bosch.

1. If classified according to the different models used, trajectory prediction methods can be divided into methods using physical models, methods using learning, and methods using planning algorithms.
2. If classified according to the information used, trajectory prediction methods can be divided into methods that use target information, methods that use dynamic target information in the environment, and methods that use static environment information.

What general algorithms are involved in trajectory prediction?

1. Intention prediction: fuzzy theory, static BNs, DBN (HMM, JumpMM), DS evidence theory, classification algorithms in machine learning.
2. Deep learning related, end-to-end output. CNN, LSTM, RNN, Attention.

#What specific information can be used for trajectory prediction?

1. Target information: current/historical speed and position information. If it is pedestrian trajectory prediction, the orientation of the pedestrian's head, joint information, gender and age information can also be used. Human attention information.
2. Dynamic target information in the environment: social force, attraction, group constraint information.
3. Static environment information: free space, map, semantic information (road structure/traffic rules/current traffic lights).

#There are currently more and more papers on trajectory prediction in academia. The main reason is that there is no effective method in the industry.

The following are industry papers:

BMW: Physics Model intent prediction (learning-based). Heuristic methods are used to integrate expert knowledge, simplify the interaction model, and add game theory ideas to the classification model of intention prediction [3].

BENZ: Mainly related papers on intent prediction, using DBN[4] .

Uber: LaneRCNN[5].

Google: VectorNet[6].

Huawei: HOME[7].

Waymo: TNT[8].

Aptive: Covernet[9].

NEC: R2P2[10].

SenseTime: TPNet[11].

Meituan: StarNet[12]. pedestrian.

Aibee: Sophie[13]. pedestrian.

MIT: Social lstm[14]. pedestrian.

USTC: STGAT[15]. pedestrian.

baidu: Lane-Attention[16].

##Apollo: You can read the following blog for reference.

​https://www.cnblogs.com/liuzubing/p/11388485.html​

Apollo’s prediction module receives perception, Positioning and map module input.

1. First, the scene was split into two scenes: ordinary cruising road and intersection.

2. Then divide the importance of the perceived targets into targets that can be ignored (will not affect your own car) and targets that need to be handled with caution (may affect your own car). rook) and normal targets (somewhere in between).

3. Then enter the Evaluator, which is essentially an intention prediction.

4. Finally enter the predictor, which is used to predict trajectory generation. Perform different operations for different scenarios such as stationary targets, driving along the road, freeMove, and intersections.

5 Dataset

(1) NGSIM

This data set is highway driving data collected by FHWA in the United States, including the driving status of all vehicles on US101, I-80 and other roads in a time period. The data is acquired using a camera and then processed into track point records one by one. Its data set is a CSV file. The data doesn't have much noise.

is more information at the overall dispatch level, such as road planning, lane setting, traffic flow adjustment, etc. The vehicle kinematic state needs to be further extracted. The processing code can be used on github below.

​https://github.com/nachiket92/conv-social-pooling​

##(2) INTERACTION

This dataset was created for the UC Berkeley Mechanical Systems Control Laboratory (MSC Lab) with collaborators from the Karlsruhe Institute of Technology (KIT) and the École Nationale Supérieure de Mines Paris (MINES ParisTech). International, adversarial, collaborative data set (INTERACTION). It can accurately reproduce a large number of interactive behaviors of road users (such as vehicles and pedestrians) in various driving scenarios in different countries.

##​http://www.interaction-dataset.com/​

(3)apolloscape

This is Apollo’s public autonomous driving data set, which contains data provided for trajectory prediction. The internal file is a 1min data sequence of 2fps. The data structure includes frame number ID, target ID, target category, position xyz, length, width, and height information, and heading. The target category includes small cars, large cars, pedestrians, bicycles/electric vehicles, and others.

​https://apolloscape.auto/trajectory.html​

(4) TRAF

This data set focuses on high-density traffic conditions. This condition can help the algorithm better focus on analyzing human driver behavior in uncertain environments. Each frame of data contains approximately 13 motor vehicles, 5 pedestrians and 2 bicycles

​https://gamma.umd.edu/researchdirections/autonomousdriving/ad​

There are many trajectory prediction projects using this dataset in the link.

(5) nuScenes

The big news is coming. This data set was proposed in April 2020. It collected 1,000 driving scenes in Boston and Singapore, two cities with heavy traffic and challenging driving conditions. Its data set has related papers, you can take a look to get a better understanding of this data set.

​https://arxiv.org/abs/1903.11027​

There are prediction-related competitions in this data set, you can pay attention to them.

​https://www.nuscenes.org/prediction?externalData=all&mapData=all&modalities=Any​

6 Evaluation Index

The currently mainly used evaluation index is the geometric metric.

Geometric measurement has many indicators, the main ones used are ADE, FDE, and MR.

ADE is the normalized Euclidean distance. FDE is the Euclidean distance between the final prediction points. MR is the miss rate. There are many different names. The main thing is to set a threshold. If the Euclidean distance between the predicted points is lower than this prediction, it will be recorded as a hit. If it is higher than this threshold, it will be recorded as a miss. Finally, a percentage will be calculated.

Geometric metric is an important indicator to measure the similarity between the predicted trajectory and the actual trajectory, and can well represent the accuracy. But for the purpose of trajectory prediction, it is meaningless to just evaluate the accuracy. There should also be probabilistic measures to evaluate uncertainty, especially for multi-modal output distributions; there should also be task-level measures, robustness measures, and efficiency evaluations.

Probability measure: KL divergence, predicted probability, and cumulative probability can be used as probability measure. For example, NLL, KDE-based NLL [17]. Task-level metrics: Evaluate the impact of trajectory prediction on back-end regulation (piADE, piFDE) [18]. Robustness: Consider the length or duration of the observed part of the trajectory before prediction; the size of the training data; input data sampling frequency and sensor noise; neural network generalization, overfitting and input utilization analysis; perception module If there is a problem with the input input, whether the function is guaranteed to be normal and other factors. Efficiency: Consider computing power.

As shown in the figure below, the main consideration of this paper is that based on the true value (blue), the purple and green trajectories predicted by the gray target car have the same ADE and FDE if geometric metrics are used, but different The prediction method will have an impact on the planning of the own car, and there is currently no such metric for evaluating the task level, so they proposed piADE and piFDE to do this.

7 Three questions

Question 1: Three different trajectory prediction methods : Where are the application scenarios of physical model-based, learning-based, and planning-based? What are the advantages and disadvantages?

Different modeling methods can combine and exploit different types of contextual information. All modeling methods can be expanded by using the target's contextual clues and dynamic and static environments. However, different modeling methods exhibit different levels of complexity and efficiency in combining different categories of semantic information.

1. Physical model-based method

Applicable scenarios: targets, static environments, and dynamic simulations can be Explicit transfer equation modeling.

Advantages:

• Physically model-based methods can be easily applied across environments by selecting appropriate transfer equations, while No training data is required, although some data for parameter estimation is useful. In the paper, a simple CV model also produces reasonable results.
• The physical model-based method can be easily extended by combining it with target agent cues.

Disadvantages:

• This approach to explicit modeling may not capture the complexity of the real world well .
• The transfer equation lacks global information in space and time, resulting in the possibility of obtaining a local optimal solution.

#Such shortcomings limit the use of physical methods to short-term prediction or obstacle-free environments.

2. Learning-based method

Applicable scenarios: Suitable for current environments with complex unknown information (such as Common areas with rich semantics), and this information can be used for a relatively large prediction range.

Advantages:

• Learning-based methods can potentially handle all types of contextual information encoded in the collected data set. Some of them are map-based, and others can be used to further extend context information.

Disadvantages:

• Needs to collect enough data at a specific location for training.
• Contextual information expansion may lead to involved learning, data efficiency, and generalization issues.
• Tends to be used in non-safety critical components. In ADAS, interpretability is more important, which cannot be achieved based on learning methods.

3. Planning-based method

Applicable scenario: The end point is determined and the environment map is Available scenes, well represented.

Advantages:

• If the above two conditions are met, it can achieve better accuracy than physical methods and better than learning-based methods The method has better generalization ability.

shortcoming:

• Traditional planning algorithms: Dijkstra, Fast Marching Method, optimal sampling-based motion planners will grow exponentially as the number of targets, the size of the environment, and the prediction range increase.
• Compared with simple physics-based models, the parameters of context-cue-based planning methods (such as reward functions for inverse programming and models for forward programming) are trivial and generally easier to learn , but in terms of reasoning, it is less efficient for high-dimensional (target) agent states.

Planning-based methods are essentially map-aware and abstacle-aware, and naturally extend using semantic clues. Typically, they encode situational complexity into the goal/reward equation, but this may not properly integrate dynamic line inputs. Therefore, the authors had to design specific modifications to incorporate dynamic input into the prediction algorithm (Jump Markov Processes, local adaptations of the predicted trajectory, game-theoretic). Unlike learning-based methods, target inputs can be easily merged because both forward and backward planning processes are based on the same target dynamic model.

Question 2: Has the problem of trajectory prediction been solved now?

#The need for trajectory prediction largely depends on the application domain and the specific use case scenarios therein. It may not be said that the problem of trajectory prediction has been solved in the short term. Take the automotive industry as an example. Because there are special standards and regulations that define maximum speeds, traffic rules, distribution of pedestrian speeds and accelerations, and specifications for comfortable acceleration/deceleration rates of vehicles, it seems to be the most powerful in formulating requirements and proposing solutions. Mature. It can be said that for the AEB function of smart cars, the solution has reached a performance level that allows industrial production of consumer products, and its required use cases have been solved. As for other use cases, more standardization and clear articulation of requirements will be needed in the near future. And robustness and stability still need to evolve.

So before answering the question of whether trajectory prediction has solved this problem, we should at least set the standard.

Currently in the field of robotics

• methods based on physical models and learning can be implemented in a short time (1-2s) Higher accuracy. It is very suitable for local motion planning and collision avoidance of crowds. The simplest CV model has a good effect on the local planning of the robot. If you consider the interaction between pedestrians and the impact of the presence of robots on pedestrian movements, there are many advanced algorithms.
• There are great challenges for global path planning that requires prediction of 15-20 seconds. The requirements can be relaxed appropriately, and it becomes very important to understand the dynamic and static context input (which affects the operation in the long term, reasoning on the environment map, and the intention inference of the target). For local and global path planning, position-independent methods are best suited for predicting motion in various environments.
• The current robot predicts that the ADE in 4.8s is 0.19-0.4m. A simple velocity model can also achieve an ADE of 0.53m. 9s predicts an ADE of 1.4-2m.

Currently in the field of autonomous driving:

• Most work considers pedestrians crossing the road : Start walking, keep walking, stop walking.
• Bicycle: A cyclist approaches an intersection with up to five different directions of road behind them.

# Question 3: Are current evaluation techniques for measuring trajectory prediction performance good enough?

There is currently a lack of systematic approaches to prediction algorithms, especially for trajectory prediction methods that consider contextual inputs and predict an arbitrary number of targets.

Now most authors only use geometric measures (AED, FDE) as an indicator to measure the quality of an algorithm. For long-term predictions, however, the predictions are often multimodal and associated with uncertainty, and performance evaluation of such methods should use metrics that take this into account, such as the negative log-likelihood or logarithm obtained from KLD loss.

There is also a need for probabilistic measures that better reflect the randomness of human motion and the uncertainty involved in perceptual imperfections.

There is also robustness evaluation, which needs to consider the stability of the system when detection errors, tracking defects, self-positioning uncertainty or map changes occur on the sensing side.

At the same time, although the currently used data sets include very comprehensive scenarios, these data sets are usually semi-automatically annotated, and therefore can only provide incomplete and noisy true value estimates. Furthermore, trajectory length is often insufficient in some application areas where long-term predictions are required. Finally, the interaction between targets in the data set is usually limited. For example, in a sparse environment, it is difficult for targets to influence each other.

In summary: In order to evaluate the prediction quality, researchers should choose more complex data sets (including non-convex obstacles, long trajectories and complex interactions) and complete metrics (geometric probability). A better method is to set different accuracy requirements based on different prediction times, different observation periods, and different scene complexities. And there should be robustness evaluation and real-time evaluation. In addition, there should be relevant indicators that can measure the impact of ADAS systems on the backend [18] and indicators that measure sensitivity to hazardous scenarios [1].

8 Future Directions

From the discussion in [2], quoted here.

The current trend is to use more complex methods to go beyond using a single model KF method

Direction:

1. Use enhanced contextual information: Deeper semantic information can be used, which should provide a better understanding of static environments. And the current use of semantic features for trajectory prediction still needs to be developed
2. Regarding socially-aware scenarios: ① Most current methods assume that the behaviors of all observed people are similar , their movements can be predicted by the same model and the same features, while the capture and reasoning of high-level social attributes are still in the early stages of development. ② Most feasible methods are based on the assumption that cooperative behavior between people, and real people may be more inclined to optimize individual goals rather than joint strategies, so methods combined with traditional AI game theory are very promising.
3. For long-term predictions, contextual information becomes particularly important as intent is considered based on context and surrounding circumstances. Many current learning-based methods treat individuals as particles and use them to learn to transfer information to determine the direction of future movement. Extending these models with more intent-driven predictions, similar to human goal-directed behavior, will benefit long-term predictions.
4. Most planning-based methods rely on a given set of goals, which makes them unusable or inappropriate without prior knowledge of the destinations or where the number of destinations may be too high. accurate. This makes automatic inference of destination endpoints based on semantic information important. Or it can dynamically identify possible destinations in the environment and perform trajectory prediction based on this. This allows planning-based methods to be used in unknown environments.
5. Current methods focus on solving a specific type of task, such as when there is an obvious movement pattern in the environment, or when the spatial structure of the environment and the destination of the target agent When known in advance. Trajectory prediction methods need to be able to adapt to undefined/changing environments and handle unexpected situations. This requires transfer learning and some methods of coping with new environments, where learning and reasoning about basic immutable rules, or general pedestrian behavior or collision avoidance is not appropriate. Domain adaptation can be used to learn generalization models.
6. Another direction to pay attention to: robustness and integrability.

To sum up: To put it simply, the context information should be used more deeply, it is best to have different behavioral models and game theory for different goals, and make more decisions based on more information. Excellent intent prediction, automatic inference of endpoints, generalization problems to new environments, robustness and integrability.

The above is the detailed content of Discuss the current status and development trends of autonomous driving trajectory prediction technology. For more information, please follow other related articles on the PHP Chinese website!