CVPR 2024 | Is there only single-person data in the synthetic video data set? M3Act solves the problem of crowd behavior labeling

The AIxiv column is a column where this site publishes academic and technical content. In the past few years, the AIxiv column of this site has received more than 2,000 reports, covering top laboratories from major universities and companies around the world, effectively promoting academic exchanges and dissemination. If you have excellent work that you want to share, please feel free to contribute or contact us for reporting. Submission email: liyazhou@jiqizhixin.com; zhaoyunfeng@jiqizhixin.com

• ## Paper link: https://arxiv. org/abs/2306.16772
• Project link: https://cjerry1243.github.io/M3Act/
• Paper title: M3Act: Learning from Synthetic Human Group Activities

Introduction

Recognizing and understanding crowd behavior through visual information is an important area in video monitoring, interactive robots, autonomous driving and other fields is one of the key technologies, but obtaining large-scale crowd behavior annotation data has become a bottleneck in the development of related research. Nowadays, synthetic datasets are becoming an emerging method to replace real-world data, but synthetic datasets in existing research mainly focus on the estimation of human pose and shape. They often only provide synthetic animation videos of

single characters, which are not suitable for video recognition tasks of crowds.

In this article, the author proposes M3Act, a synthetic data generation framework suitable for multi-group crowd behavior. Experiments show that this synthetic data set can greatly improve the performance of downstream models in multi-person tracking and group activity recognition, and can replace more than 62.5% of real data on the DanceTrack task, thereby reducing data annotation costs in real-world application scenarios. Additionally, this synthetic data framework proposes a new class of tasks: controllable 3D swarm activity generation. This task aims to directly control the swarm activity generation results using multiple inputs (activity category, swarm size, trajectory, density, speed, and text input). The authors rigorously define tasks and metrics and provide competitive baselines and results.

Data generation

Based on the Unity engine development, M3Act covers a variety of behavioral types of crowd data , provides highly diverse and realistic video images, as well as comprehensive data labeling. Compared to other synthetic datasets, M3Act provides more comprehensive labeled data, including 2D and 3D markers as well as fine-grained individual-level and group-level labels, thus making it an ideal synthesis to support multi-person and multi-group research tasks Dataset generator.

Data generator includes 25 3D scenes, 104 high dynamic range panoramic images, 5 light settings, 2200 character models, 384 animations (14 action categories ) and 6 group activity types. The data generation process is as follows. First, all parameters within a simulation scenario are determined through a randomization process, and then a 3D scene with background objects, lights and cameras, and a group of character models with animation are generated based on the parameters. Finally, RGB images are rendered from multiple viewpoints and the labeled results are exported.

To ensure a high degree of diversity in simulated data, M3Act provides randomization for nearly all aspects of the data generation process. This includes the number of groups in the scene, the number of people in each group, the position of the group, the arrangement of people in the group, the position of the individuals, the textures of the instantiated characters, as well as the scene, lighting conditions, camera position, characters, group activity, atoms Selection of action and animation clips. Each group activity is also built as a parameterized module. These parameters include the number of individuals in the swarm and the specific atomic actions allowed within the swarm's activity.

The final generated data set is divided into two parts. The first part "M3ActRGB" contains 6000 simulations of single but multiple types of group activities and 9000 simulations of multiple groups and multiple types, with a total of 6 million RGB images and 48 million bounding boxes. The second part "M3Act3D" contains only 3D data. It consists of more than 65,000 150-frame simulations of a single multi-type group activity, totaling 87.6 hours. To the authors' knowledge, M3Act3D's group size and interaction complexity are significantly higher than previous multiplayer sports datasets, making it the first large-scale 3D dataset for large group activities.

Experimental results

The actual effect of M3Act is through three The core experiments demonstrate: multi-person tracking, group activity recognition and controllable group activity generation.

Experiment 1: Multi-person Tracking

Research findings , after adding synthetic data to the training of the existing model MOTRv2 [1], the model has significantly improved on all 5 indicators, especially jumping from 10th to 2nd in the ranking on the HOTA indicator. At the same time, when 62.5% of the real data in the training set was replaced by synthetic data, the model could still achieve similar performance. In addition, compared to other synthetic data sources, such as BEDLAM and GTA-Humans, M3Act provides greater performance improvements for model training, indicating that it is more suitable for multi-person group activity tasks. Finally, the table below shows the training results of different models under M3Act. The results show that M3Act is effective in various models.

Experiment 2: Group activity recognition

Similarly , M3Act also improves the performance of two existing group activity recognition models, as shown in the following table: As the amount of synthetic data used for pre-training increases, the recognition accuracy continues to improve. When using 100% synthetic data, the accuracy of the group activity recognition model Composer [2] increased by an average of 4.87% at the group level and 7.43% at the individual level, while another group activity recognition model Actor Transformer [3] improved at the group level. An increase of 5.59% in accuracy was seen on , and an increase of 5.43% at the individual level.

The following table shows the group recognition accuracy on CAD2 and Volleyball (VD) using different input modalities. Performance gains in experiments demonstrate that M3Act's synthetic data can effectively benefit downstream tasks and span different models, input modalities, and datasets.

Experiment 3: Controllable 3D group activity generation

The author proposes a new type of task: controllable 3D group activity generation. The task aims to synthesize a set of 3D human actions from Gaussian noise based on a given activity class label and an arbitrary population size. Although existing studies can generate multi-player actions, they are limited to two-person scenarios or groups with a fixed number of people. Therefore, the authors propose two baseline methods. In the first baseline approach, group activity is implemented by repeatedly invoking the single-person motion diffusion model MDM [4], so the generation process for each individual is independent. The second method adds an interactive transformer (IFormer) based on MDM. Due to its modeling of human interactions, MDM+IFormer is able to produce coordinated group activities in a single forward pass.

The author considers the following evaluation indicators at both the group and individual levels: recognition accuracy, Frechette initial distance (FID), diversity and multimodality. In addition, based on the social force model, the author adds four location-based indicators at the group level: collision frequency, repulsive interaction force, contact repulsive force, and total repulsive force. The results show:

• MDM+IFormer is able to generate group activities with well-aligned character positions. See qualitative graph below.
• Both baseline methods can generate diverse activities matching the input conditions, but MDM+IFormer achieves better FID scores.
• Interaction transformers in MDM+IFormer greatly reduce the frequency of collisions within generated group activities.

in conclusion

The authors of the paper demonstrate the advantages of M3Act through three core experiments of multi-modality and enhanced performance, as well as the introduction of a new generation task. In experiments on multi-person tracking and group activity recognition, they observed that the model's generalization ability to unseen test cases improved as more synthetic data was added.

In addition, the synthetic data in M3Act can replace real data in some target fields without affecting performance, which is expected to reduce the need for a large amount of real data during the training process, thereby reducing cost of data collection and annotation. This finding demonstrates the potential of small or even zero samples to migrate from simulated data to real-world data.

In the generation of controllable 3D group activities, although MDM+IFormer is only the baseline model for this task, it still learns the interaction rules of character movement and under control Generate well-aligned group activity. Notably, although generative approaches currently outperform procedural approaches, they demonstrate the potential to control group actions directly from a variety of signals (activity category, group size, trajectory, density, speed, and text input). As data availability increases and generative model capabilities improve in the future, the authors predict that generative methods will eventually gain dominance and become more widely used in social interactions and collective human activities.

Although the complexity of group behavior in the M3Act dataset may be limited by the heuristic rules used in the data generation process, M3Act provides significant flexibility in incorporating new group activities , thereby adapting to any specific downstream task. These new populations can originate from expert-guided heuristic rules, rules generated by large language models, or the output of controllable 3D population activity generation models. Furthermore, the paper's authors recognize the domain differences that exist between synthetic and real-world data. With the addition of assets in the data generator in future releases, it will be possible to improve the model's generalization capabilities and mitigate these differences.

^{[1] Yuang Zhang, Tiancai Wang, and Xiangyu Zhang. Motrv2: Bootstrapping end-to-end multi-object tracking by pretrained object detectors . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22056–22065, 2023.}

^{##[2] Honglu Zhou, Asim Kadav, Aviv Shamsian, Shijie Geng, Farley Lai, Long Zhao, Ting Liu, Mubbasir Kapadia, and Hans Peter Graf. Composer: Compositional reasoning of group activity in videos with keypoint-only modality. Proceedings of the 17th European Conference on Computer Vision (ECCV 2022 ), 2022.}

^{[3] Kirill Gavrilyuk, Ryan Sanford, Mehrsan Javan, and Cees GM Snoek. Actor-transformers for group activity recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 839–848, 2020.}

[4] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209.14916, 2022.

The above is the detailed content of CVPR 2024 | Is there only single-person data in the synthetic video data set? M3Act solves the problem of crowd behavior labeling. For more information, please follow other related articles on the PHP Chinese website!