ImageNet zero-sample accuracy exceeds 80% for the first time, and the strongest open source CLIP model on the planet is updated

Although ImageNet has long completed its historical mission, it is still a key data set in the field of computer vision.

In 2016, the sota accuracy rate of the classification model trained on ImageNet was still less than 80%; to this day, zero-shot generalization based on large-scale pre-trained models alone can achieve an accuracy of 80.1%.

##Laion recently trained a brand new ViT-G/14 CLIP## using the open source OpenCLIP framework # Model, on the ImageNet data set, the accuracy of the original OpenAI CLIP is only 75.4%, while OpenCLIP achieves a zero-shot accuracy of 80.1%, and achieves a zero-shot of 74.9% on MS COCO Image retrieval (Recall@5), which is also currently the most powerful open source CLIP model.

LAION stands for Large-scale Artificial Intelligence Open Network. It is a non-profit organization with members from all over the world, aiming to provide large-scale Artificial Intelligence to the public. Scale machine learning models, datasets, and associated code. They claim to be truly Open AI, 100% non-profit and 100% free.

Interested friends can update the CLIP model at hand!

Model address: https://huggingface.co/laion/CLIP-ViT-bigG-14 -laion2B-39B-b160k

The specific performance of the OpenCLIP model on each data set is shown in the table below.

Zero-shot capability

Generally speaking, computer vision (CV) models perform better in various tasks The sota performance on is based on training data in specific fields and cannot be generalized to other fields or tasks, resulting in a limited understanding of the general attributes of the visual world.

The generalization problem is especially important in fields that lack large amounts of training data.

Ideally, a CV model should learn the semantic content of images rather than focusing too much on specific labels in the training set. For example, for an image of a dog, the model should be able to understand that there is a dog in the image, and further understand that there are trees in the background, the time is daytime, the dog is on the grass, etc.

But the current results obtained by "classification training" are exactly opposite to expectations. The model learns to push the dog's internal representation into the same "dog vector space" and the cat into the same "Cat vector space", the answer to all questions is binary, that is, whether the image can be aligned with a category label.

Retraining a classification model for new tasks is also a solution, but the training itself requires a lot of time and financial investment. to collect classification datasets and train models.

Fortunately, OpenAI’s CLIP model is a very flexible classification model and can usually be used in new classification tasks without retraining.

Why CLIP can Zero-Shot

Contrastive Language-Image Pretraining (CLIP, Contrastive Language-Image Pretraining) is a model released by OpenAI in 2021, mainly based on Transformer.

CLIP consists of two models, a Transformer encoder to convert text to embedding, and a Visual Transformer (ViT) to encode images.

Both text and image models within CLIP are optimized during pre-training to align similar text and images in vector space. During the training process, image-text pairs in the data are pushed closer together in the vector space, while image and text vectors that do not belong to a pair are separated.

There are several differences between CLIP and general classification models:

First, OpenAI uses crawled data from the Internet The benefits of training on a very large-scale data set containing 400 million text-image pairs are:

1. CLIP training only requires "image-text pairs" and does not require specific class labels, and this Types of data are abundant in today’s social media-centric online world.

2. Large datasets mean CLIP’s ability to understand common textual concepts in images.

3. The text descriptor often contains various features in the image, not just a category feature, which means that a more comprehensive image and text representation can be established.

The above advantages are also a key factor in CLIP's ability to establish Zero-shot. The author of the paper also compared the ResNet-101 model and CLIP model specially trained on ImageNet, and applied it to other data derived from ImageNet. Set, the figure below shows the performance comparison.

It can be seen that although ResNet-101 is trained on ImageNet, its performance on similar datasets is better than CLIP Performance on the same task is much worse.

When applying the ResNet model to other fields, a common method is "linear probe", that is, the characteristics learned in the last few layers of the ResNet model are input into a linear classifier, and then Fine-tuned for specific data sets.

In the CLIP paper, linear detection ResNet-50 is compared with zero-shot CLIP. The conclusion is that in the same scenario, zero-shot CLIP performs better than ResNet in multiple tasks. Linear detection in -50.

However, it is worth noting that when given more training samples, Zero-shot does not outperform linear detection.

Use CLIP for Zero-shot classification

As you can know from the above description, the image and text encoder can create a 512-dimensional vector that maps the input image and text input to the same vector space.

Using CLIP for zero-shot classification means putting category information into text sentences.

For example, if you input an image and want to determine whether its category is a car, bird or cat, you can create three text strings to represent the category:

T1 represents car: a photo of a car

T2 represents a bird: a photo of a bird

T3 represents a cat: a photo of a cat

Enter the category description into the text encoder, and A vector can be obtained that can represent the category.

Assume that the input is a photo of a cat. Use the ViT model to encode it. After obtaining the image vector, calculate the cosine distance between it and the category vector as the similarity. If the similarity with T3 is the highest, it means The category of the image belongs to cats.

As you can see, the category label is not a simple word, but based on the template 「a photo of a {label}」 The format of is rewritten into a sentence, which can be extended to category prediction without training restrictions.

In the experiment, using this prompt template improved the classification accuracy of ImageNet by 1.3 percentage points. However, the prompt template does not always improve performance. In actual use, it needs to be tested based on different data sets.

Python implementation

It is also very easy to quickly use CLIP for zero-shot classification. The author selected the frgfm/imagenette data set in Hugging Face as a demonstration. The data set contains 10 labels. , and all are saved as integer values.

Using CLIP for classification requires converting integer value labels into corresponding text content.

#Before directly calculating the similarity between tags and photos, you need to initialize the CLIP model, which can be implemented using the CLIP found through Hugging Face transformers.

The text transformer cannot read text directly, but requires a set of integer values ​​called token IDs (or input_IDs), where each A unique integer represents a word or sub-word (ie token).

Input the converted tensor into the text transformer to obtain the text embedding of the label

Note that currently CLIP The output vector has not been normalized, and the similarity results obtained after dot multiplication are inaccurate.

Next, you can select an image in the data set for testing, and obtain the image vector after the same processing process.

After converting the image into a vector with dimensions of (1, 3, 224, 224), input it into the model to obtain embedding

The next step is to calculate the dot product similarity between the image embedding and the ten label text embeddings in the data set. The one with the highest score is the predicted category.

The model gives the result for the cassette player (cassette player), in the entire data set After running it again, we can get an accuracy of 98.7%

In addition to Zero-shot classification, multi-modal search, target detection, generative Models such as OpenAI's Dall-E and Stable disusion, CLIP open a new door to computer vision.

The above is the detailed content of ImageNet zero-sample accuracy exceeds 80% for the first time, and the strongest open source CLIP model on the planet is updated. For more information, please follow other related articles on the PHP Chinese website!