Beihang University breaks down modal barriers and introduces a universal physical counterattack method across visible and infrared modes.

In recent years, the exploration of safety assessment of visual perception systems has gradually deepened. Researchers have successfully implemented visible light modal safety assessment technology based on different carriers such as glasses, stickers, clothes, etc., and there are also some new attempts targeting infrared modalities. . But they can only work in a single mode.

With the development of artificial intelligence technology, visible light-thermal infrared imaging technology has been used in many safety-critical tasks such as security monitoring and autonomous driving. Among them, visible light imaging can It provides rich texture information during the day, and infrared imaging can clearly display the thermal radiation distribution of the target at night. The combination of the two brings many advantages to the visual perception system, such as 24-hour full coverage and freedom from environmental limitations. Therefore, a unified security assessment method for multi-modal visual perception systems is also urgently needed to be studied.

However, achieving multimodal assessment is extremely challenging. First of all, it is difficult to universally apply attack methods under different imaging mechanisms. Previous methods were proposed based on the imaging characteristics of specific target modalities, and are difficult to work in other modalities. Furthermore, it is difficult to balance stealth performance, production cost and flexible application. It is not easy to be dual effective in visible light and the more difficult infrared mode, and it is even more difficult to achieve low-cost and convenient production and use.

Faced with many challenges, researchers from the Beihang Institute of Artificial Intelligence explored the common shape attributes between visible light and infrared modes, and innovatively proposed "cross- Modal universal countermeasure patch" to achieve visible-infrared synchronized stealth. It selects materials that are easy to obtain, low cost, and have excellent thermal insulation properties to make convenient patches, which are ready to use. While filling the gaps in the robustness evaluation technology of visible light-infrared multi-modal detection systems in the current physical world, it also takes into account The ease and immediacy of physical implementation. Experiments demonstrate the effectiveness of this method under different detection models and modalities, as well as its generalization in multiple scenarios. Currently, this paper has been accepted by ICCV 2023.

Paper link: https://arxiv.org/abs/2307.07859

Code link: https://github.com/Aries-iai/Cross-modal_Patch_Attack

Technical points

This research uses evolutionary algorithm as The basic framework is based on the three perspectives of shape modeling, shape optimization, and modal balance for program design and effect improvement. The specific process is as shown in the figure:

##1. Multi-anchor shape modeling based on spline interpolation

For the basics For the shape modeling part, the researchers designed a new paradigm of point optimization modeling, which can directly adjust the patch shape by changing the point coordinates. In this process, the movement of the anchor point will not be restricted by direction, distance, etc., effectively increasing the search for patch shapes. space. On this basis, in order to ensure the naturalness of the shape, it also uses the spline interpolation method to achieve smooth connections, and the splines will follow the control points more closely.

2. Boundary-limited shape optimization algorithm based on differential evolution

Effective optimization means are required to implement the attack. This researcher considered the time cost, actual effect, etc., used the evolutionary algorithm as the basic framework, and improved it from the two perspectives of boundary setting and fitness function:

( 1) Boundary setting: Boundary setting for anchor points improves the effectiveness of deformation and reduces time costs. It has the following settings: no loops or self-intersections will be formed in the curve segment; cusps will not easily appear in the curve segment; and they will not appear in the invalid area.

Take the anchor point

as an example. The blue part in the figure below is the boundary setting legend, and the orange part is the error instance:

About the boundary determination of the anchor point

The mathematical expression is as follows:

(2) Fitness function: Unlike previous work that only targets a single mode for attack evaluation, this work focuses on the two modes of visible light and infrared, and there is a natural problem of balancing the differences in modal effects. . Therefore, in order to avoid going to the extreme of easily optimizing a single mode, the researchers innovatively proposed a cross-modal fitness function based on detector confidence score perception, encouraging the exploration of successful directions while balancing the difference in the effects of the two modes, and finally the survival of the fittest based on scores. Taking into account the difference in attack difficulty between the initial stage and the later stage, it uses an exponential function instead of a linear function to highlight the difference in attack progress in different stages.

##                                                                                                                                                                   ’ ’ s to ’ s ’ ’ s ’ s ’ s ’ s ‐ ‐ ‐ ‐ ‐ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​

The algorithm iterates the exploration process until both modes successfully attack, and outputs the optimal shape strategy.The complete optimization process is as follows:

##Experimental results

Experiment 1: Cross-modal attack performance verification for different series of detectors

Experiment 2: Shape ablation experiment

##Experiment 3: Ablation experiment for cross-modal fitness function

Experiment 4: Method robustness verification under physical implementation deviation

Experiment 5: Validation of method effectiveness under different physical conditions

##Performance verification visualization results under different angles, distances, postures, and scenarios

##Summary

##The work of this paper is based on natural shape optimization. Combining deformation patches with cross-modal attacks, a visible-infrared multi-modal robustness evaluation method in physical environments is designed. This method can evaluate the robustness of a multi-modal (visible light-infrared) target detection system, effectively correct the detector model based on the evaluation results, and simultaneously improve the accuracy of target image detection in both visible light and infrared modes. In physics It can be truly implemented and applied in the environment, and contribute to the robustness evaluation and improvement of multi-modal detection systems.

The above is the detailed content of Beihang University breaks down modal barriers and introduces a universal physical counterattack method across visible and infrared modes.. For more information, please follow other related articles on the PHP Chinese website!