Here's how to teach ChatGPT how to read pictures

The "Wen Sheng Tu" model will be popular in 2022, so what will be popular in 2023?

The answer from machine learning engineer Daniel Bourke is: the other way around!

No, a newly released "pictures and text" model has exploded on the Internet, and its excellent effects have caused many netizens to repost and like it.

is not only a basic "look at pictures and speak" function, but also can write love poems, explain plots, design dialogues for objects in pictures, etc., this AI can do all Hold it firmly!

For example, when you find tempting food on the Internet, just send it the picture, and it will immediately identify the required ingredients and cooking steps:

Even some of Leeuwenhoek’s details in the picture can be “seen” clearly.

When asked how to get out of the upside-down house in the picture, AI's answer was: Isn't there a slide on the side?

This new AI is called BLIP-2 (Bootstrapping Language-Image Pre-training 2), and the code is currently open source.

The most important thing is that, unlike previous research, BLIP-2 uses a universal pre-training framework, so it can be connected to your own language model arbitrarily.

Some netizens are already imagining the powerful combination after changing the interface to ChatGPT.

Steven Hoi, one of the authors, even said: BLIP-2 will be the "multi-modal version of ChatGPT" in the future.

So, what other magical places are there in BLIP-2? Look down together.

First-class understanding ability

The gameplay of BLIP-2 can be said to be very diverse.

You only need to provide a picture, and you can talk to it, and it can meet various requirements such as telling stories, reasoning, and generating personalized text.

For example, BLIP-2 can not only easily identify the scenic spot in the picture as the Great Wall, but also introduce the history of the Great Wall:

The Great Wall of China was built by Qin Shihuang in 221 BC to protect the imperial capital. Built to protect against invasion from the north.

Give it a movie still, BLIP-2 not only knows where it comes from, but also knows the ending of the story: the sinking of the Titanic, male The Lord drowned.

BLIP-2 also grasps the human expression very accurately.

When asked what the man's expression in this picture was and why he was like this, BLIP-2's answer was: he was afraid of the chicken because it was flying towards him.

What’s even more amazing is that BLIP-2 also performs very well on many open questions.

Let it write a romantic sentence based on the picture below:

Its answer is this: Love is like a sunset, It's hard to see it coming, but when it happens, it's so beautiful.

Not only does this person have perfect understanding, but he also has strong literary attainments!

Let it generate a dialogue for the two animals in the picture. BLIP-2 can also easily handle the arrogant cat x silly cute dog Settings:

Cat: Hey, dog, can I ride on your back?

Dog: Of course, why not?

Cat: I'm tired of walking in the snow.

So, how does BLIP-2 achieve such a powerful understanding ability?

Implementing new SOTA on multiple visual language tasks

Considering that the end-to-end training cost of large-scale models is getting higher and higher, BLIP-2 uses a general and efficient pre-training method. Training strategy:

Bootstrap visual language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models.

This also means that everyone can choose the model they want to use.

In order to bridge the gap between modes, the researcher proposed a lightweight query Transformer.

The Transformer is pre-trained in two stages:

The first stage guides visual language representation learning from the frozen image encoder, and the second stage guides vision from the frozen language model to language generation study.

In order to test the performance of BLIP-2, the researchers started from zero-sample image-text generation, visual question answering, image-text retrieval, and image subtitles respectively. It was evaluated on the task.

The final results show that BLIP-2 achieved SOTA on multiple visual language tasks.

Among them, BLIP-2 is 8.7% higher than Flamingo 80B on zero-shot VQAv2, and the training parameters are reduced by 54 times.

And it is obvious that a stronger image encoder or a stronger language model will produce better performance.

It is worth mentioning that the researcher also mentioned at the end of the paper that BLIP-2 still has a shortcoming, that is, the lack of context learning ability :

Each sample contains only one image-text pair, and it is currently impossible to learn the correlation between multiple image-text pairs in a single sequence.

Research Team

The research team of BLIP-2 comes from Salesforce Research.

The first author is Junnan Li, who is also the author of BLIP, which was launched a year ago.

is currently a senior research scientist at Salesforce Asia Research Institute. Graduated from the University of Hong Kong with a bachelor's degree and a Ph.D. from the National University of Singapore.

The research field is very wide, including self-supervised learning, semi-supervised learning, weakly supervised learning, and visual-language.

The following is the paper link and GitHub link of BLIP-2. Interested friends can pick it up~

​Paper link: https://arxiv.org/pdf/2301.12597. pdf

GitHub link: https://github.com/salesforce/LAVIS/tree/main/projects/blip2

Reference link: [1]https://twitter.com/mrdbourke /status/1620353263651688448

[2]​https://twitter.com/LiJunnan0409/status/1620259379223343107

The above is the detailed content of Here's how to teach ChatGPT how to read pictures. For more information, please follow other related articles on the PHP Chinese website!