How much potential do fixed-parameter models have? Hong Kong Chinese, Shanghai AI Lab and others proposed an efficient video understanding framework EVL

Visual basic models have achieved remarkable development in the past two years. On the one hand, pre-training based on large-scale Internet data has preset a large number of semantic concepts for the model, thus having good generalization performance; but on the other hand, in order to make full use of the model size brought by large-scale data sets Growth makes related models face inefficiency problems when migrating to downstream tasks, especially for video understanding models that need to process multiple frames.

• Paper link: https://arxiv.org/abs/2208.03550
• Code link: https://github.com/OpenGVLab/efficient-video-recognition

Based on the above two characteristics , researchers from the Chinese University of Hong Kong, Shanghai Artificial Intelligence Laboratory and other institutions proposed an efficient video understanding transfer learning framework EVL. By fixing the weight of the backbone basic model, it saves training calculations and memory consumption; at the same time, by utilizing multi-level, Fine-grained intermediate features maintain the flexibility of traditional end-to-end fine-tuning as much as possible.

Figure 1 below shows the results of the EVL method on the video understanding dataset Kinetics-400. Experiments show that this method saves training overhead while still fully exploring the potential of the basic visual model in video understanding tasks.

Figure 1: Kinetics-400 recognition accuracy comparison, the horizontal axis is the amount of inference calculation, and the vertical axis is the accuracy.

Method

The overall schematic diagram of the algorithm is shown in Figure 2(a). For a video sample, we take T frames and input them into an image recognition network (taking CLIP as an example) and extract features. Compared with traditional methods, we extract multi-layer, unpooled features from the last few layers of the image recognition network to obtain richer and finer-grained image information; and the parameter weights of the image recognition network are always consistent in video learning. Stay fixed. Subsequently, the multi-layer feature maps are sequentially input into a Transformer decoder for video-level information aggregation. The multi-layer decoded [CLS] features are used to generate the final classification prediction.

As shown in Figure 2(b), due to the disorder when the Transformer decoder aggregates features, we added additional temporal information modeling modules to the network to better Extract location-related fine-grained timing information. Specifically, we add three additional types of position-related timing information: the first is the temporal position embeddings (Position Embeddings), the second is the temporal dimension depth-separable convolution (Depthwise Convolution), and the third is the attention between adjacent frames force information. For inter-frame attention information, we extract the Query and Key features of the corresponding layer from the image recognition network, and calculate the attention map between adjacent frames (different from the image recognition network, the attention map is composed of the Query from the same frame and Key features are obtained). The resulting attention map can explicitly reflect the position changes of objects between adjacent frames. After linear projection, the attention map obtains a vector group that reflects the object's displacement characteristics, and is integrated into the image features in the form of element-by-element addition.

Figure 2: EVL algorithm structure diagram. (a) Overall structure, (b) Sequential information modeling module.

##Figure 3: Inter-frame attention features mathematical expression.

Experiment

In Figure 1 and Table 1, we quoted some important methods in previous video understanding. Despite focusing on reducing training overhead, our method still outperforms existing methods in terms of accuracy (with the same amount of computation).

In Table 2 we show the reduction in training overhead brought by the fixed backbone network. In terms of memory, on the V100 16GB GPU, the fixed backbone network can enable a single-card batch size to reach a maximum of 64, while end-to-end training can only reach 8; in terms of time, the fixed backbone network can save 3 to 4 times the training time.

In Table 3 we show the improvement of recognition performance by fine-grained feature maps. The multi-layer unpooled features allow us to maintain a considerable degree of flexibility when fixing the backbone network weights. Using unpooled features brings the most significant improvement (about 3%), followed by using multi-layer decoders and mid-layer features, which also bring about 1% performance improvement each.

Finally we show the effect of the fine-grained timing information module in Table 4. Although fine-grained timing information has a limited impact on the performance of Kinetics-400, they are very important for the performance of Something-Something-v2: the three fine-grained timing information modules bring a total of about 0.5% and about 14% performance improvement.

Table 1: Comparison results with existing methods on Kinetics-400

Table 2: Training overhead reduction caused by fixed backbone network weights

Table 3: The impact of fine-grained feature maps on accuracy

Table 4: The effect of fine-grained time series information modeling on different data sets

Summary

This article proposes the EVL video understanding learning framework, which for the first time demonstrates the great potential of a fixed image backbone network in video understanding problems, and also makes high-performance video understanding more friendly to research groups with limited computing resources. We also believe that as the quality and scale of visual basic models improve, our method can provide a reference for subsequent research on lightweight transfer learning algorithms.

The above is the detailed content of How much potential do fixed-parameter models have? Hong Kong Chinese, Shanghai AI Lab and others proposed an efficient video understanding framework EVL. For more information, please follow other related articles on the PHP Chinese website!