Adding real agent behavior to the CARLA autonomous driving simulator

arXiv paper "Insertion of real agents behaviors in CARLA autonomous driving simulator", June 22, Spain.

The role of simulation in autonomous driving is becoming increasingly important due to the need for rapid prototyping and extensive testing. Physics-based simulation offers multiple advantages and benefits at a reasonable cost while eliminating risks to prototyping, drivers and Vulnerable Road Users (VRUs). However, there are two main limitations. First, the well-known reality gap refers to the difference between reality and simulation that prevents simulated autonomous driving experiences from achieving effective real-world performance. Second, there is a lack of empirical knowledge about the behavior of real agents (humans), including back-up drivers or passengers and other road users such as vehicles, pedestrians or cyclists.

Agent simulations are usually pre-programmed, probabilistically random, or generated based on real data, but do not represent the behavior of real agents interacting with specific simulation scenarios. This paper proposes a preliminary framework that enables real-time interaction between real agents and simulated environments, including autonomous vehicles, and generates synthetic sequence data from simulated sensors across multiple views for training predictive systems that rely on behavioral models.

This approach integrates immersive virtual reality (VR) and human motion capture (HMC) systems with CARLA for autonomous driving. It describes the hardware and software architecture and discusses the so-called behavioral gap.

As shown in the figure is an overview of the method: (1) CARLA-UE is equipped with head (VR headset) and body (motion capture system) poses. (2) Generate scenes including autonomous vehicles and digital pedestrians. (3) Provide environment for pedestrians (via VR headsets). (4) Automatic vehicle sensors sense the environment, including pedestrians.

The following introduces the functions of the immersive VR system in the CARLA autonomous driving simulator.

Achieve full pedestrian immersion by conducting behavioral and interaction studies using the capabilities provided by UE4 and external hardware such as VR glasses and a set of motion sensors.

The CARLA open source simulator is implemented on UE4, which provides high-quality rendering, realistic physics environments and an ecosystem of interoperable plug-ins. CARLA simulates dynamic traffic scenes and provides an interface between the virtual world created by UE4 and the road agent running in the scene. CARLA is designed as a server-client system to achieve this, with the server running the simulation and rendering the scene. Communication between client and server is done through sockets.

The main characteristics of inserting real agent behavior into the simulation are based on five points (as shown in the figure): 1)

Avatar Control

: CARLA blueprint library , collect the architecture of all characters and attributes, modify the pedestrian blueprint, and create an immersive and movable VR interface between people and the virtual world; 2) Body Tracking: Use a set of inertial sensors and proprietary external software Capture subject movement and motion perception through real scenes, and integrate Avatar's movement into the simulator through .bvh files; 3) Sound Design: Since CARLA is an audio-less simulator, positional sounds are introduced in the environment , enhance the sense of immersion; 4) eHMI integration: Realize the communication of autonomous vehicle status and intention information, and do interactive research; 5) Scene simulation: Design traffic scenes in the CARLA client , control the behavior of vehicles and pedestrians. Communication between road users is an important factor in the traffic environment. In the experiment, an external human-machine interface (eHMI) was provided for the autonomous vehicle to communicate its status and intentions to actual road users.

As shown in the figure, the proposed eHMI design consists of a light strip along the front of the car. The picture on the left is without eHMI, and the picture on the right is with eHMI. This allows studying the impact of the interface on decision-making when pedestrian trajectories converge with vehicle trajectories in virtual scenes.

The entire system architecture is shown in the figure:

CARLA provides different options to simulate traffic and specific traffic scenarios. Populate simulations with realistic urban traffic conditions with traffic management modules. The control of each vehicle is executed in a specific thread. Communication with other layers is managed through synchronous messaging.

Control traffic flow by setting parameters that force specific behaviors. For example, cars can be allowed to speed, ignore traffic light conditions, ignore pedestrians, or force lane changes.

The main body is integrated into the simulator including a 3-D model map of the city. Each map is based on an OpenDRIVE file describing a fully annotated road layout. This feature allows to design your own maps, reproduce the same traffic scenarios in real and virtual environments, evaluate the integration of the real behavior of the simulator, and be able to conduct field studies by comparing interaction results.

The hardware settings are as shown in the figure: During the experiment, Oculus Quest 2 was used as a head-mounted device (HMD), created by Meta, 6GB RAM processor, two adjustable 1832 x 1920 lenses, 90Hz refresh rate and 256 GB of RAM. The Quest 2 features Wi-Fi 6, Bluetooth 5.1 and USB Type-C connectivity, SteamVR support and 3D speakers. For full-body tracking, package the solution with PNS with an inertial tracker. The kit includes a standalone VR headset, 2 motion controllers, 17 Studio inertial body sensors, 14 sets of straps, a charging case and a Studio transceiver.

Since CARLA build and Quest 2 only rely on Windows, VR Immersion System currently relies on UE4.24 and Windows 10 operating systems. Using the TCP sockets plugin, the editor's position of all participants and other useful parameters are sent from the Python API, integrating each participant's voice or the eHMI of an autonomous vehicle. "VR Preview" starts the game on HMD. Perception Neuron Studio works with Axis Studio, which supports up to 3 subjects at a time and 23 body and finger sensors simultaneously.

The picture shows the simulation of interactive traffic conditions: (a) 3D world design. (b) Pedestrian matching executor Avatar. (c) Self-driving cars. (d) Environmental sounds and intelligent sounds. (e) eHMI. (f) Traffic lights and traffic signs.

The above is the detailed content of Adding real agent behavior to the CARLA autonomous driving simulator. For more information, please follow other related articles on the PHP Chinese website!