search
HomeTechnology peripheralsAI2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the list

Self-supervised learning enables computers to observe the world and understand it by learning the structure of images, speech, or text. This has driven many of the recent major advances in artificial intelligence.

Despite the considerable efforts that researchers around the world have invested in this area, there are currently large differences in the way self-supervised learning algorithms learn from images, speech, text and other modalities. Therefore, the artificial intelligence forum Analytics India Magazine launches the top ten self-supervised learning models in 2022 for the readers. ​

Data2vec

2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the list

Paper link: https://arxiv.org/pdf/2202.03555.pdf

Open source code: https://t.co/3x8VCwGI2x pic.twitter.com/Q9TNDg1paj

Meta AI released the data2vec algorithm in January for speech, image and text related computer vision model. According to the AI ​​team, the model is highly competitive in NLP tasks.

It does not use contrastive learning or reconstruction that relies on input examples. The Meta AI team stated that the training method of data2vec is to represent the predictive model by providing a partial view of the input data.

The team said: "We first encode the masked training samples in the student model. After that, in the same model, we encode the unmasked input samples to build the training target. This model (teacher model) and the student model only differ in parameters."

2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the list

This model predicts the model representation of unmasked training samples based on the masked training samples. This eliminates the dependence on modality-specific objectives in the learning task.

ConvNext

2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the list

Paper link: https://arxiv.org/pdf/2201.03545.pdf

Open source code: https://t.co/nWx2KFtl7X

ConvNext, also called ConvNet model for the 2020s, is a model released by the Meta AI team in March Model. It is entirely based on ConvNet's modules and is therefore accurate, simple in design, and scalable.

2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the list

##VICReg

2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the list##Paper link: https:// t.co/H7crDPHCHV

Open source code: https://t.co/oadSBT61P3

Variance-invariant covariance regularization (VICReg) combines the variance terms and Decorrelation mechanism based on redundancy reduction and covariance regularization to avoid the collapse of the encoder producing constant or uninformative vectors.

2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the listVICReg does not require techniques such as weight sharing between branches, batch normalization, feature normalization, output quantization, stopping gradients, memory banks, etc., and performs well on several downstream tasks The results achieved are comparable to the state of the art. Furthermore, it has been experimentally demonstrated that the variance regularization term can stabilize the training of other methods and promote performance improvements.

STEGO

2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the listPaper link: https://arxiv.org/abs/2203.08414

MIT’s Computer Science and Artificial Intelligence Laboratory collaborated with Microsoft and Cornell University to develop the Self-supervised Transformer for Energy-Based Graph Optimization (STEGO) to solve one of the most difficult tasks in computer vision. : Assign a label to every pixel of an image without human supervision.

2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the list#STEGO learned "semantic segmentation" - simply put, assigning a label to each pixel in the image.

Semantic segmentation is an important skill for today's computer vision systems because images may be interfered by objects. To make matters more difficult, these objects don't always fit within the text box. Algorithms are often better suited to discrete “things” like people and cars than to hard-to-quantify things like vegetation, the sky, and mashed potatoes.

Take the scene of dogs playing in the park as an example. Previous systems may only be able to identify dogs, but by assigning a label to each pixel of the image, STEGO can decompose the image into several main components: Dog , sky, grass and its owner.

2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the list

Machines that can "see the world" are crucial to a variety of emerging technologies, such as self-driving cars and predictive models for medical diagnosis. Since STEGO can learn without labels, it can detect objects in different domains, even objects that humans do not yet fully understand.

CoBERT

2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the list

Paper link: https://arxiv.org/pdf/2210.04062.pdf

For self-supervised speech representation learning, researchers from the Chinese University of Hong Kong (Shenzhen) proposed Code BERT (CoBERT). Unlike other self-distillation methods, their model predicts representations from different modalities. The model converts speech into a sequence of discrete codes for representation learning.

2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the list

First, the research team used the HuBERT pre-trained code model to train in discrete space. They then refined the code model into a speech model, aiming to perform better learning across modalities. The significant improvement on the ST task suggests that CoBERT's representations may carry more linguistic information than previous work.

CoBERT outperforms the performance of the best current algorithms on ASR tasks and brings significant improvements in the SUPERB Speech Translation (ST) task.

FedX

2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the list

## Paper link: https://arxiv.org/abs/2207.09158

FedX is an unsupervised federated learning framework launched by Microsoft in cooperation with Tsinghua University and Korea Advanced Institute of Science and Technology. Through local and global knowledge extraction and comparative learning, the algorithm learns unbiased representations from discrete and heterogeneous local data. Furthermore, it is an adaptable algorithm that can be used as an add-on module to various existing self-supervised algorithms in federated learning scenarios.

TriBYOL

2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the list

Paper link: https://arxiv.org/pdf/2206.03012.pdf

Hokkaido University in Japan proposed TriBYOL for small-batch self-supervised representation learning. Under this model, researchers do not require large amounts of computing resources to learn good representations. This model has a triplet network structure and combines a three-view loss, thereby improving efficiency and outperforming several self-supervised algorithms on multiple data sets.

2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the list

ColloSSL

2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the list##Paper link: https://arxiv.org/pdf/ 2202.00758.pdf

Researchers at Nokia Bell Labs, in collaboration with Georgia Institute of Technology and the University of Cambridge, have developed ColloSSL, a collaborative self-supervised algorithm for human activity recognition.

2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the list Unlabeled sensor data sets captured simultaneously by multiple devices can be viewed as natural transformations of each other, which then generate signals for representation learning. This paper proposes three methods - device selection, contrastive sampling and multi-view contrastive loss.

LoRot

2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the list

Paper link: https://arxiv.org/pdf/2207.10023.pdf

Sungkyunkwan A university research team proposes a simple self-supervised auxiliary task that predicts localizable rotations (LoRot) with three attributes to assist in supervising the target.

2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the list

This model has three major characteristics. First, the research team guided the model to learn rich features. Second, distributed training does not change significantly while the self-supervision transition occurs. Third, the model is lightweight and versatile and has high adaptability to previous technologies.

2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the list

TS2Vec

2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the list

##Paper link: https:// arxiv.org/pdf/2106.10466.pdf

Microsoft and Peking University proposed a general learning framework TS2Vec for representation learning of time series at any semantic level. The model performs contrastive learning in a hierarchical technique in an enhanced context view, providing a strong context representation for individual timestamps.

2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the list

The results show that the TS2Vec model achieves significant improvements in performance compared to state-of-the-art unsupervised time series representation learning.

In 2022, there will be huge innovations in the two fields of self-supervised learning and reinforcement learning. Although researchers have been debating which is more important, as self-supervised learning guru Yann LeCun said: "Reinforcement learning is like the cherry on the cake, supervised learning is the icing on the cake, and self-supervised learning is the cake itself. 》

Reference materials:

https://analyticsindiamag.com/top-10-self-supervised-learning-models-in-2022/

The above is the detailed content of 2022 Top10 self-supervised learning models released! Eight achievements of the United States and China dominate the list. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:51CTO.COM. If there is any infringement, please contact admin@php.cn delete
Gemma Scope: Google's Microscope for Peering into AI's Thought ProcessGemma Scope: Google's Microscope for Peering into AI's Thought ProcessApr 17, 2025 am 11:55 AM

Exploring the Inner Workings of Language Models with Gemma Scope Understanding the complexities of AI language models is a significant challenge. Google's release of Gemma Scope, a comprehensive toolkit, offers researchers a powerful way to delve in

Who Is a Business Intelligence Analyst and How To Become One?Who Is a Business Intelligence Analyst and How To Become One?Apr 17, 2025 am 11:44 AM

Unlocking Business Success: A Guide to Becoming a Business Intelligence Analyst Imagine transforming raw data into actionable insights that drive organizational growth. This is the power of a Business Intelligence (BI) Analyst – a crucial role in gu

How to Add a Column in SQL? - Analytics VidhyaHow to Add a Column in SQL? - Analytics VidhyaApr 17, 2025 am 11:43 AM

SQL's ALTER TABLE Statement: Dynamically Adding Columns to Your Database In data management, SQL's adaptability is crucial. Need to adjust your database structure on the fly? The ALTER TABLE statement is your solution. This guide details adding colu

Business Analyst vs. Data AnalystBusiness Analyst vs. Data AnalystApr 17, 2025 am 11:38 AM

Introduction Imagine a bustling office where two professionals collaborate on a critical project. The business analyst focuses on the company's objectives, identifying areas for improvement, and ensuring strategic alignment with market trends. Simu

What are COUNT and COUNTA in Excel? - Analytics VidhyaWhat are COUNT and COUNTA in Excel? - Analytics VidhyaApr 17, 2025 am 11:34 AM

Excel data counting and analysis: detailed explanation of COUNT and COUNTA functions Accurate data counting and analysis are critical in Excel, especially when working with large data sets. Excel provides a variety of functions to achieve this, with the COUNT and COUNTA functions being key tools for counting the number of cells under different conditions. Although both functions are used to count cells, their design targets are targeted at different data types. Let's dig into the specific details of COUNT and COUNTA functions, highlight their unique features and differences, and learn how to apply them in data analysis. Overview of key points Understand COUNT and COU

Chrome is Here With AI: Experiencing Something New Everyday!!Chrome is Here With AI: Experiencing Something New Everyday!!Apr 17, 2025 am 11:29 AM

Google Chrome's AI Revolution: A Personalized and Efficient Browsing Experience Artificial Intelligence (AI) is rapidly transforming our daily lives, and Google Chrome is leading the charge in the web browsing arena. This article explores the exciti

AI's Human Side: Wellbeing And The Quadruple Bottom LineAI's Human Side: Wellbeing And The Quadruple Bottom LineApr 17, 2025 am 11:28 AM

Reimagining Impact: The Quadruple Bottom Line For too long, the conversation has been dominated by a narrow view of AI’s impact, primarily focused on the bottom line of profit. However, a more holistic approach recognizes the interconnectedness of bu

5 Game-Changing Quantum Computing Use Cases You Should Know About5 Game-Changing Quantum Computing Use Cases You Should Know AboutApr 17, 2025 am 11:24 AM

Things are moving steadily towards that point. The investment pouring into quantum service providers and startups shows that industry understands its significance. And a growing number of real-world use cases are emerging to demonstrate its value out

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
1 months agoBy尊渡假赌尊渡假赌尊渡假赌
R.E.P.O. Best Graphic Settings
1 months agoBy尊渡假赌尊渡假赌尊渡假赌
Will R.E.P.O. Have Crossplay?
1 months agoBy尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

ZendStudio 13.5.1 Mac

ZendStudio 13.5.1 Mac

Powerful PHP integrated development environment

VSCode Windows 64-bit Download

VSCode Windows 64-bit Download

A free and powerful IDE editor launched by Microsoft

MantisBT

MantisBT

Mantis is an easy-to-deploy web-based defect tracking tool designed to aid in product defect tracking. It requires PHP, MySQL and a web server. Check out our demo and hosting services.

SecLists

SecLists

SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.