Using CNN as the basic model, the deformable convolution InternImage achieves a new record in detection and segmentation!

The booming development of large-scale visual Transformers in recent years has pushed the performance boundaries in the field of computer vision. Vision Transformer models defeat convolutional neural networks by expanding the number of model parameters and training data. Researchers from Shanghai Artificial Intelligence Laboratory, Tsinghua University, Nanda, SenseTime, and Hong Kong Chinese summarized the gaps between convolutional neural networks and visual Transformers. From an operator level, traditional CNNs operators lack long-distance dependencies and adaptive spatial aggregation capabilities; from a structural level, traditional CNNs structures lack advanced components.

In response to the above technical problems, Researchers from Pujiang Laboratory, Tsinghua University and other institutions innovatively proposed a large-scale model based on convolutional neural networks, called For InternImage, it uses sparse dynamic convolution as the core operator and achieves adaptive spatial aggregation by inputting relevant information as a condition. InternImage enables learning more powerful and robust large-scale parameter patterns from massive data by reducing the strict inductive bias of traditional CNNs. Its effectiveness has been verified on visual tasks including image classification, object detection and semantic segmentation. It has achieved competitive results in challenging benchmark data sets including ImageNet, COCO and ADE20K. At the same parameter level, it has surpassed the visual Transformer structure and provided a new direction for large image models.

• ## Paper link: https://arxiv.org/abs/2211.05778
• Open source code: https://github.com/OpenGVLab/InternImage

##Limitations of traditional convolutional neural networks

Expanding the scale of the model is an important strategy to improve the quality of feature representation. In the field of computer vision, the expansion of the model parameters can not only effectively enhance the depth The model has the representation learning ability and can realize learning and knowledge acquisition from massive data. ViT and Swin Transformer expanded the depth model to 2 billion and 3 billion parameter levels for the first time. The classification accuracy of their single models in the ImageNet data set also exceeded 90%, far exceeding the traditional CNN network and small-scale models, breaking through the technical bottleneck. . However, due to the lack of long-distance dependency and spatial relationship modeling capabilities, traditional CNN models cannot achieve model scale expansion capabilities similar to the Transformer structure. The researchers summarized the differences between traditional convolutional neural networks and visual Transformer:

(1) From the operator level, the multi-head attention mechanism of visual Transformer has long-distance dependencies And adaptive spatial aggregation capabilities, benefiting from this, the visual Transformer can learn from massive data more powerful and robust representations than CNN networks.

(2) From the perspective of model architecture, in addition to the multi-head attention mechanism, the visual Transformer has more advanced modules that the CNN network does not have, such as Layer Normalization (LN), feedforward Neural network FFN, GELU, etc.

Although some recent work attempts to use large kernel convolutions to obtain long-distance dependencies, they are still far from the state-of-the-art visual Transformers in terms of model scale and accuracy.

Further expansion of deformable convolutional networks

InternImage improves the scalability of convolutional models and alleviates inductive bias by redesigning operators and model structures , including (1) DCNv3 operator, which introduces shared projection weights, multi-group mechanisms and sampling point modulation based on the DCNv2 operator. (2) Basic module, integrating advanced modules as the basic module unit for model construction (3) Module stacking rules, standardizing the width, depth, number of groups and other hyper-parameters of the model when expanding the model.

This work is dedicated to building a CNN model that can effectively scale to large-scale parameters. First, the deformable convolution operator DCNv2 is redesigned to adapt to long-distance dependencies and weaken inductive bias; then, the adjusted convolution operator is combined with advanced components to establish a basic unit module; finally, explore and implement Stacking and scaling rules of modules to build a base model with large-scale parameters and powerful representations can be learned from massive data.

At the operator level, this study first summarizes the main differences between the convolution operator and other mainstream operators. The current mainstream Transformer series models mainly rely on the multi-head self-attention mechanism to achieve large model construction. Its operators have long-distance dependencies, which are sufficient to construct connection relationships between long-distance features, and also have spatial adaptive aggregation capabilities to achieve pixel-level construction. Relationship. However, this global attention mechanism has huge computing and storage requirements, making it difficult to achieve efficient training and rapid convergence. Likewise, local attention mechanisms lack long-range feature dependence. Large-core dense convolution has no spatial aggregation ability, so it is difficult to overcome the natural inductive bias of convolution, which is not conducive to expanding the model. Therefore, InternImage designs dynamic sparse convolution operators to achieve global attention effects without wasting too much computing and storage resources, achieving efficient training.

Based on the DCNv2 operator, the researchers redesigned, adjusted and proposed the DCNv3 operator. Specific improvements include the following parts.

(1) Shared projection weight. Similar to conventional convolution, different sampling points in DCNv2 have independent projection weights, so its parameter size is linearly related to the total number of sampling points. In order to reduce parameter and memory complexity, we draw on the idea of ​​separable convolution and use position-independent weights to replace the grouping weights. The projection weights are shared between different sampling points, and all sampling position dependencies are retained.

(2) Introduce multi-group mechanism. Multi-group design was first introduced in grouped convolution and is widely used in Transformer's multi-head self-attention. It can be paired with adaptive spatial aggregation to effectively improve the diversity of features. Inspired by this, researchers divide the spatial aggregation process into several groups, and each group has an independent sampling offset. Since then, different groups of a single DCNv3 layer have different spatial aggregation patterns, resulting in rich feature diversity.

(3) Sampling point modulation scalar normalization. In order to alleviate the instability problem when the model capacity is expanded, the researchers set the normalization mode to Softmax normalization on a sample-by-sample basis. This not only makes the training process of large-scale models more stable, but also constructs a model of all sampling points. connection relationship.

After building the DCNv3 operator, you first need to standardize the overall details of the basic modules and other layers of the model, and then explore the Stacking strategy to build InternImage. Finally, models with different parameter amounts are constructed according to the expansion rules of the proposed model.

Basic module. Different from the bottleneck structure widely used in traditional CNN, this study adopts a basic module closer to ViTs, equipped with more advanced components, including GELU, layer normalization (LN) and feed-forward network (FFN), which have been Proven to be more efficient in a variety of vision tasks. The details of the basic module are shown in the figure above, where the core operator is DCNv3, which predicts the sampling bias and modulation scale by passing the input features through a lightweight separable convolution. For other components, follow the same design as a normal Transformer.

Overlay rules. In order to clarify the block stacking process, this study proposes two module stacking rules. The first rule is the number of channels in the last three stages, which is determined by the number of channels in the first stageDecision, that is, ; The second rule is that the group number of each module corresponds to the number of channels in each stage, that is, ; The third, stacking The mode is fixed to "AABA", that is, the number of module stacks in stages 1, 2 and 4 is the same , and not greater than that in stage 3 . Therefore, a model with a parameter volume of 30M is chosen as the basis. The specific parameters are: the number of Steam output channels is 64; the number of groups is 1/16 of the number of input channels in each stage. The number of module stacks in stages 1, 2, and 4 is 4, the number of module stacks in stage 3 is 18, and the model parameters are 30M.

Model scaling rules. Based on the optimal model under the above constraints, this study normalized two scaling dimensions of the network model: depth D (number of module stacks) and width C (number of channels), using the restriction factor and scale the depth and width along the composite coefficient , i.e., , where , according to experiments the optimal setting is .

Following this rule, this study constructed models of different scales, namely InternImage-T, S, B, L, and XL. The specific parameters are:

Experimental results

Image classification experiment: By using 427M public data sets: Laion-400M, YFCC15M, CC12M, InternImage-H achieves an accuracy of 89.2% on ImageNet-1K.

Object detection: Taking the largest InternImage-H as Backbone network, and using DINO as the basic detection framework, pre-trained the DINO detector on the Objects365 data set, and then fine-tuned on COCO. The model achieved an optimal result of 65.4% in the target detection task, breaking through the performance boundary of COCO target detection.

Semantic Segmentation: On semantic segmentation, InternImage-H It also achieved very good performance, and combined with Mask2Former achieved the current highest of 62.9% on ADE20K.

Conclusion

This study proposes InternImage, a new CNN-based large-scale basic model that can Powerful representations are provided for versatile vision tasks such as image classification, object detection, and semantic segmentation. The researchers adjusted the flexible DCNv2 operator to meet the needs of the basic model, and developed a series of blocking, stacking and scaling rules based on the core operator. Extensive experiments on object detection and semantic segmentation benchmarks have verified that InternImage can achieve equivalent or better performance than well-designed large-scale visual Transformers trained on large amounts of data, indicating that CNN is also a considerable step in large-scale visual basic model research. choose. Still, large-scale CNNs are still in their early stages of development, and the researchers hope InternImage can serve as a good starting point.

The above is the detailed content of Using CNN as the basic model, the deformable convolution InternImage achieves a new record in detection and segmentation!. For more information, please follow other related articles on the PHP Chinese website!