search
HomeTechnology peripheralsAIYou can learn 'without matching data'! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR

Multimodal contrastive representation (MCR) aims to encode inputs from different modalities into a semantically aligned shared space

As the visual-linguistic domain With the great success of the CLIP model, more and more modal contrast representations have begun to emerge and have achieved significant improvements in many downstream tasks, but these methods rely heavily on large-scale and high-quality paired data

In order to solve this problem, researchers from Zhejiang University and other institutions proposed Concatenated Multimodal Contrast Representation (C-MCR), a multimodal contrastive representation learning method that does not require paired data and is extremely efficient in training.

You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR

Please click the following link to view the paper: https://arxiv.org/abs/2305.14381

C-MCR project homepage link: https://c-mcr.github.io/C-MCR/

Model and code address: https://github.com/MCR -PEFT/C-MCR

This method connects different pre-trained contrastive representations through hub modalities without using any paired data, and we learn powerful audio-visual and 3D point cloud-text representation, and achieved SOTA effects on multiple tasks such as audio-visual retrieval, sound source localization, and 3D object classification.

Introduction

Multimodal contrastive representation (MCR) aims to map data from different modalities into a unified semantic space. With the great success of CLIP in the visual-linguistic field, learning contrastive representations between more modal combinations has become a hot research topic, attracting more and more attention.

However, the generalization ability of existing multimodal contrastive representations mainly benefits from a large number of high-quality data pairs. This severely limits the development of contrastive representations on modalities that lack large-scale high-quality data. For example, the semantic correlation between audio and visual data pairs is often ambiguous, and paired data between 3D point clouds and text are scarce and difficult to obtain.

However, we have observed that these modal combinations that lack paired data often have a large amount of high-quality paired data with the same intermediate modality. For example, in the audio-visual domain, although the quality of audiovisual data is unreliable, there is a large amount of high-quality paired data between audio-text and text-visual.

Similarly, while the availability of 3D point cloud-text pairing data is limited, 3D point cloud-image and image-text data are abundant. These hub modes can establish further links between modes.

Considering that modalities with a large amount of paired data often already have pre-trained contrastive representations, this article directly attempts to connect the contrastive representations between different modalities through the hub modality. Thus, a new contrastive representation space is constructed for modal combinations lacking paired data.

Using Concatenated Multimodal Contrast Representation (C-MCR), you can build connections with a large number of existing multimodal contrastive representations through overlapping modes, thereby learning a wider range of modalities. alignment relationship between them. This learning process does not require any paired data and is extremely efficient

C-MCR has two key advantages:

The focus is on flexibility:

The ability of C-MCR is to provide contrasting representations that lack direct pairing for modal learning. From another perspective, C-MCR treats each existing multi-modal contrast representation space as a node, and overlapping modes as key hub modes

By connecting individual isolated multimodal contrastive representations, we are able to flexibly extend the obtained multimodal alignment knowledge and mine a wider range of intermodal contrastive representations

2. Efficiency:

Since C-MCR only needs to build connections for the existing representation space, it only needs to learn two simple mappers. Training parameters and training costs are extremely low.

In this experiment, we use text as a hub to compare visual-text (CLIP) and text-audio (CLAP) to represent spatial connections, and finally get high-quality The visual-audio representation

Similarly, by comparing the image-connected text-visual (CLIP) and the visual-3D point cloud (ULIP) to represent the space, a set of 3D Point Cloud-Text Contrast Representation

Method

Figure 1 (a) Introduces the algorithm flow of C-MCR (taking the use of text to connect CLIP and CLAP as an example) .

You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR

The data of text (overlapping modalities) are encoded into text features by the text encoders of CLIP and CLAP respectively:,.

At the same time, there is also a large amount of unpaired single-modal data that is encoded into CLIP and CLAP spaces respectively, forming image memory and audio memory

Feature semantic enhancement refers to the process of improving and optimizing features to enhance their semantic expression capabilities. By appropriately adjusting the features, it can more accurately reflect the meaning to be expressed, thereby improving the effect of language expression. Feature semantic enhancement technology has important application value in the field of natural language processing, which can help machines understand and process text information, and improve machine capabilities in semantic understanding and semantic generation

We can start by improving the semantic information of representation to enhance the robustness and comprehensiveness of spatial connections. In this regard, we first discuss it from the two perspectives of semantic consistency and semantic integrity

Inter-modal semantic consistency

CLIP and CLAP have learned reliable aligned image-text and text-audio representations respectively.

We exploit this inherent modal alignment in CLIP and CLAP to generate image and audio features that are semantically consistent with the i-th text, thereby better quantifying contrast in the representation space Modality gap and more direct mining of correlations between non-overlapping modalities:

You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR

Intra-modal semantic integrity

Different representation spaces will have different tendencies for the semantic expression of data, so the same text in different spaces will inevitably have semantic deviations and losses. This semantic bias is accumulated and amplified when connecting representation spaces.

To enhance the semantic completeness of each representation, we propose to add zero-mean Gaussian noise to the representations and renormalize them to the unit hypersphere:

You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR

As shown in Figure 1 (c), in the contrastive representation space, each representation can be seen to represent a point on the unit hypersphere. Adding Gaussian noise and renormalizing allows the representation to represent a circle on the unit sphere.

When the spatial distance between two features is closer, their semantic similarity is higher. Therefore, the features within the circle have similar semantics, and the circle can express the semantics more completely

2. Alignment of Inter-MCR

After representation semantic enhancement, we use two mappers and to remap CLIP and CLAP representations to a new shared space

You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR

The new space needs to ensure that semantically similar representations from different spaces are close to each other.

(You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR,You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR) derived from the same text are naturally semantically consistent and can be regarded as real tag pairs, while those derived from (You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR,You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR) of (You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR,You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR) can be regarded as pseudo-label pairs . The semantics between

(You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR,You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR) are highly consistent, but the connections learned from them are not suitable for audio -Indirectly speaking. Although the semantic consistency of the (You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR,You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR) pair is less reliable, it is more directly beneficial to audio-visual representation.

In order to connect the two contrastive representation spaces more comprehensively, we simultaneously align (You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR,You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR) and (You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR,You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR):

You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR

##3. Alignment of Intra-MCR

In addition to the connection between spaces, there is also a modality gap phenomenon within the contrast representation space. That is, in the contrastive representation space, although the representations of different modalities are semantically aligned, they are distributed in completely different subspaces. This means that the more stable connections learned from (You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR,You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR) may not be well inherited by audio-visual.

To solve this problem, we propose to realign the different modal representations of each contrastive representation space. Specifically, we remove the negative example exclusion structure in the contrast loss function to derive a loss function for reducing the modality gap. A typical contrastive loss function can be expressed as:

You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR

After we eliminate the negative pair exclusion term, the final formula can be simplified as:

You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR

Experiment

Experimentally, we obtained by using text to connect audio-text space (CLAP) and text-visual space (CLIP) Audio-visual representation, using image connection 3D point cloud-image space (ULIP) and image-text space (CLIP) to obtain 3D point cloud-text representation.

The results of zero-sample audio image retrieval on AVE and Flickr-SoundNet are as follows:

You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR

The zero-sample sound source localization results on MUSIC-Solo and VGGSS are as follows:

You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR

The zero-shot counterfactual audio image recognition results on Ex-VGGSS and Ex-FlickrNet are as follows:

You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR

zero- on ModelNet40 The shot 3D point cloud classification results are as follows:

You can learn without matching data! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR

The above is the detailed content of You can learn 'without matching data'! Zhejiang University and others proposed connecting multi-modal contrast representation C-MCR. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:51CTO.COM. If there is any infringement, please contact admin@php.cn delete
The AI Skills Gap Is Slowing Down Supply ChainsThe AI Skills Gap Is Slowing Down Supply ChainsApr 26, 2025 am 11:13 AM

The term "AI-ready workforce" is frequently used, but what does it truly mean in the supply chain industry? According to Abe Eshkenazi, CEO of the Association for Supply Chain Management (ASCM), it signifies professionals capable of critic

How One Company Is Quietly Working To Transform AI ForeverHow One Company Is Quietly Working To Transform AI ForeverApr 26, 2025 am 11:12 AM

The decentralized AI revolution is quietly gaining momentum. This Friday in Austin, Texas, the Bittensor Endgame Summit marks a pivotal moment, transitioning decentralized AI (DeAI) from theory to practical application. Unlike the glitzy commercial

Nvidia Releases NeMo Microservices To Streamline AI Agent DevelopmentNvidia Releases NeMo Microservices To Streamline AI Agent DevelopmentApr 26, 2025 am 11:11 AM

Enterprise AI faces data integration challenges The application of enterprise AI faces a major challenge: building systems that can maintain accuracy and practicality by continuously learning business data. NeMo microservices solve this problem by creating what Nvidia describes as "data flywheel", allowing AI systems to remain relevant through continuous exposure to enterprise information and user interaction. This newly launched toolkit contains five key microservices: NeMo Customizer handles fine-tuning of large language models with higher training throughput. NeMo Evaluator provides simplified evaluation of AI models for custom benchmarks. NeMo Guardrails implements security controls to maintain compliance and appropriateness

AI Paints A New Picture For The Future Of Art And DesignAI Paints A New Picture For The Future Of Art And DesignApr 26, 2025 am 11:10 AM

AI: The Future of Art and Design Artificial intelligence (AI) is changing the field of art and design in unprecedented ways, and its impact is no longer limited to amateurs, but more profoundly affecting professionals. Artwork and design schemes generated by AI are rapidly replacing traditional material images and designers in many transactional design activities such as advertising, social media image generation and web design. However, professional artists and designers also find the practical value of AI. They use AI as an auxiliary tool to explore new aesthetic possibilities, blend different styles, and create novel visual effects. AI helps artists and designers automate repetitive tasks, propose different design elements and provide creative input. AI supports style transfer, which is to apply a style of image

How Zoom Is Revolutionizing Work With Agentic AI: From Meetings To MilestonesHow Zoom Is Revolutionizing Work With Agentic AI: From Meetings To MilestonesApr 26, 2025 am 11:09 AM

Zoom, initially known for its video conferencing platform, is leading a workplace revolution with its innovative use of agentic AI. A recent conversation with Zoom's CTO, XD Huang, revealed the company's ambitious vision. Defining Agentic AI Huang d

The Existential Threat To UniversitiesThe Existential Threat To UniversitiesApr 26, 2025 am 11:08 AM

Will AI revolutionize education? This question is prompting serious reflection among educators and stakeholders. The integration of AI into education presents both opportunities and challenges. As Matthew Lynch of The Tech Edvocate notes, universit

The Prototype: American Scientists Are Looking For Jobs AbroadThe Prototype: American Scientists Are Looking For Jobs AbroadApr 26, 2025 am 11:07 AM

The development of scientific research and technology in the United States may face challenges, perhaps due to budget cuts. According to Nature, the number of American scientists applying for overseas jobs increased by 32% from January to March 2025 compared with the same period in 2024. A previous poll showed that 75% of the researchers surveyed were considering searching for jobs in Europe and Canada. Hundreds of NIH and NSF grants have been terminated in the past few months, with NIH’s new grants down by about $2.3 billion this year, a drop of nearly one-third. The leaked budget proposal shows that the Trump administration is considering sharply cutting budgets for scientific institutions, with a possible reduction of up to 50%. The turmoil in the field of basic research has also affected one of the major advantages of the United States: attracting overseas talents. 35

All About Open AI's Latest GPT 4.1 Family - Analytics VidhyaAll About Open AI's Latest GPT 4.1 Family - Analytics VidhyaApr 26, 2025 am 10:19 AM

OpenAI unveils the powerful GPT-4.1 series: a family of three advanced language models designed for real-world applications. This significant leap forward offers faster response times, enhanced comprehension, and drastically reduced costs compared t

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

WebStorm Mac version

WebStorm Mac version

Useful JavaScript development tools

mPDF

mPDF

mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

EditPlus Chinese cracked version

EditPlus Chinese cracked version

Small size, syntax highlighting, does not support code prompt function

DVWA

DVWA

Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software

SublimeText3 English version

SublimeText3 English version

Recommended: Win version, supports code prompts!