You can learn "without matching data"! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR

Multimodal contrastive representation (MCR) aims to encode inputs from different modalities into a semantically aligned shared space

As the visual-linguistic domain With the great success of the CLIP model, more and more modal contrast representations have begun to emerge and have achieved significant improvements in many downstream tasks, but these methods rely heavily on large-scale and high-quality paired data

In order to solve this problem, researchers from Zhejiang University and other institutions proposed Concatenated Multimodal Contrast Representation (C-MCR), a multimodal contrastive representation learning method that does not require paired data and is extremely efficient in training.

Please click the following link to view the paper: https://arxiv.org/abs/2305.14381

C-MCR project homepage link: https://c-mcr.github.io/C-MCR/

Model and code address: https://github.com/MCR -PEFT/C-MCR

This method connects different pre-trained contrastive representations through hub modalities without using any paired data, and we learn powerful audio-visual and 3D point cloud-text representation, and achieved SOTA effects on multiple tasks such as audio-visual retrieval, sound source localization, and 3D object classification.

Introduction

Multimodal contrastive representation (MCR) aims to map data from different modalities into a unified semantic space. With the great success of CLIP in the visual-linguistic field, learning contrastive representations between more modal combinations has become a hot research topic, attracting more and more attention.

However, the generalization ability of existing multimodal contrastive representations mainly benefits from a large number of high-quality data pairs. This severely limits the development of contrastive representations on modalities that lack large-scale high-quality data. For example, the semantic correlation between audio and visual data pairs is often ambiguous, and paired data between 3D point clouds and text are scarce and difficult to obtain.

However, we have observed that these modal combinations that lack paired data often have a large amount of high-quality paired data with the same intermediate modality. For example, in the audio-visual domain, although the quality of audiovisual data is unreliable, there is a large amount of high-quality paired data between audio-text and text-visual.

Similarly, while the availability of 3D point cloud-text pairing data is limited, 3D point cloud-image and image-text data are abundant. These hub modes can establish further links between modes.

Considering that modalities with a large amount of paired data often already have pre-trained contrastive representations, this article directly attempts to connect the contrastive representations between different modalities through the hub modality. Thus, a new contrastive representation space is constructed for modal combinations lacking paired data.

Using Concatenated Multimodal Contrast Representation (C-MCR), you can build connections with a large number of existing multimodal contrastive representations through overlapping modes, thereby learning a wider range of modalities. alignment relationship between them. This learning process does not require any paired data and is extremely efficient

C-MCR has two key advantages:

The focus is on flexibility:

The ability of C-MCR is to provide contrasting representations that lack direct pairing for modal learning. From another perspective, C-MCR treats each existing multi-modal contrast representation space as a node, and overlapping modes as key hub modes

By connecting individual isolated multimodal contrastive representations, we are able to flexibly extend the obtained multimodal alignment knowledge and mine a wider range of intermodal contrastive representations

2. Efficiency:

Since C-MCR only needs to build connections for the existing representation space, it only needs to learn two simple mappers. Training parameters and training costs are extremely low.

In this experiment, we use text as a hub to compare visual-text (CLIP) and text-audio (CLAP) to represent spatial connections, and finally get high-quality The visual-audio representation

Similarly, by comparing the image-connected text-visual (CLIP) and the visual-3D point cloud (ULIP) to represent the space, a set of 3D Point Cloud-Text Contrast Representation

Method

Figure 1 (a) Introduces the algorithm flow of C-MCR (taking the use of text to connect CLIP and CLAP as an example) .

The data of text (overlapping modalities) are encoded into text features by the text encoders of CLIP and CLAP respectively:,.

At the same time, there is also a large amount of unpaired single-modal data that is encoded into CLIP and CLAP spaces respectively, forming image memory and audio memory

Feature semantic enhancement refers to the process of improving and optimizing features to enhance their semantic expression capabilities. By appropriately adjusting the features, it can more accurately reflect the meaning to be expressed, thereby improving the effect of language expression. Feature semantic enhancement technology has important application value in the field of natural language processing, which can help machines understand and process text information, and improve machine capabilities in semantic understanding and semantic generation

We can start by improving the semantic information of representation to enhance the robustness and comprehensiveness of spatial connections. In this regard, we first discuss it from the two perspectives of semantic consistency and semantic integrity

Inter-modal semantic consistency

CLIP and CLAP have learned reliable aligned image-text and text-audio representations respectively.

We exploit this inherent modal alignment in CLIP and CLAP to generate image and audio features that are semantically consistent with the i-th text, thereby better quantifying contrast in the representation space Modality gap and more direct mining of correlations between non-overlapping modalities:

Intra-modal semantic integrity

Different representation spaces will have different tendencies for the semantic expression of data, so the same text in different spaces will inevitably have semantic deviations and losses. This semantic bias is accumulated and amplified when connecting representation spaces.

To enhance the semantic completeness of each representation, we propose to add zero-mean Gaussian noise to the representations and renormalize them to the unit hypersphere:

As shown in Figure 1 (c), in the contrastive representation space, each representation can be seen to represent a point on the unit hypersphere. Adding Gaussian noise and renormalizing allows the representation to represent a circle on the unit sphere.

When the spatial distance between two features is closer, their semantic similarity is higher. Therefore, the features within the circle have similar semantics, and the circle can express the semantics more completely

2. Alignment of Inter-MCR

After representation semantic enhancement, we use two mappers and to remap CLIP and CLAP representations to a new shared space

The new space needs to ensure that semantically similar representations from different spaces are close to each other.

(,) derived from the same text are naturally semantically consistent and can be regarded as real tag pairs, while those derived from (,) of (,) can be regarded as pseudo-label pairs . The semantics between

(,) are highly consistent, but the connections learned from them are not suitable for audio -Indirectly speaking. Although the semantic consistency of the (,) pair is less reliable, it is more directly beneficial to audio-visual representation.

In order to connect the two contrastive representation spaces more comprehensively, we simultaneously align (,) and (,)：

##3. Alignment of Intra-MCR

In addition to the connection between spaces, there is also a modality gap phenomenon within the contrast representation space. That is, in the contrastive representation space, although the representations of different modalities are semantically aligned, they are distributed in completely different subspaces. This means that the more stable connections learned from (,) may not be well inherited by audio-visual.

To solve this problem, we propose to realign the different modal representations of each contrastive representation space. Specifically, we remove the negative example exclusion structure in the contrast loss function to derive a loss function for reducing the modality gap. A typical contrastive loss function can be expressed as:

After we eliminate the negative pair exclusion term, the final formula can be simplified as:

Experiment

Experimentally, we obtained by using text to connect audio-text space (CLAP) and text-visual space (CLIP) Audio-visual representation, using image connection 3D point cloud-image space (ULIP) and image-text space (CLIP) to obtain 3D point cloud-text representation.

The results of zero-sample audio image retrieval on AVE and Flickr-SoundNet are as follows:

The zero-sample sound source localization results on MUSIC-Solo and VGGSS are as follows:

The zero-shot counterfactual audio image recognition results on Ex-VGGSS and Ex-FlickrNet are as follows:

zero- on ModelNet40 The shot 3D point cloud classification results are as follows:

The above is the detailed content of You can learn "without matching data"! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR. For more information, please follow other related articles on the PHP Chinese website!