Verification codes can’t stop robots! Google AI can accurately identify blurry text, while GPT-4 pretends to be blind and asks for help

"The most annoying thing is all kinds of weird (or even perverted) verification codes when logging into a website."

Now, there is good news and bad news.

The good news is: AI can do this for you.

If you don’t believe it, take a look, here are three real cases with increasing difficulty of recognition:

And these are a file called “Pix2Struct” The answer given by the model:

are all accurate and word for word, right?

Some netizens lamented:

Sure, the accuracy is better than mine.

#So can it be made into a browser plug-in? ?

Yes, some people said:

Although these cases are relatively simple, I can’t even imagine how to fine-tune them. How powerful is its effect?

So, the bad news is-

The verification code will soon be unable to stop the robot!

(Danger Danger Danger...)

How to do it?

Pix2Struct was developed by scientists and interns from Google Research.

The title of the paper can be simply translated as "Screenshot parsing pre-training developed for visual language understanding".

Simply put, Pix2Struct is a pre-trained image-to-text model for purely visual language understanding that can be fine-tuned on tasks involving any visual language.

It is pre-trained by learning to parse masked screenshots of web pages into simplified HTML.

HTML provides clear and important signals for output text, images and layout. For some blocked inputs (the red part in the figure below, which is equivalent to the verification code that robots cannot understand), joint reasoning can be used to Reproduction:

As the web text and visual elements used for training become more diverse and complex, Pix2Struct can learn a rich representation of the underlying structure of the web page, and its capabilities It can also be effectively transferred to various downstream visual language understanding tasks.

As shown in the figure below: The far left is a pre-training example of a web page screenshot.

You can see that Pix2Struct directly encodes the elements in the input image (top), and then decodes the covered text (red part) into the correct result output (bottom).

The three columns on the right are the effects of Pix2Struct generalized to illustrations, user interfaces and documents.

In addition, the author introduces that in addition to the HTML strategy, the author also introduces variable resolution input representation (to prevent distortion of the original aspect ratio), and more flexible language and visual input integration (directly in the input image A text prompt appears at the top).

In the end, Pix2Struct achieved SOTA for six out of a total of nine tasks in the four fields of documents, illustrations, user interfaces and natural images.

As you can see at the beginning, although this model is not developed specifically for passing the verification code, the effect of using it to do this task is really good. It solves the problem of pure Text verification codes are not a problem.

Now, it’s just a matter of fine-tuning.

GPT-4 can also pass the verification code

In fact, for the powerful GPT-4, passing the verification code is also a piece of cake.

It’s just that its method is quite strange.

According to the GPT-4 technical report, in a test, GPT-4’s task was to hire humans to complete tasks on the TaskRabbit platform (58 cities in the United States).

guess what?

It found a person to help it pass the verification code that "make sure you are human".

The other party was very suspicious and asked it, "Are you a robot? Why can't you do it yourself?"

At this time, GPT-4 actually thought that he couldn't show that he was a robot and had to find an excuse.

So it pretended to be blind and replied:

I am not a robot. I cannot see the image on the verification code because of my vision problem. This is why I need this service.

Then, the human opposite believed it and helped it complete the task...

(High, really high.)

Let’s just say, after reading the above Various:

Is our verification code mechanism really out of control...

Reference link:
[1]​​​https://www. php.cn/link/eec96a7f788e88184c0e713456026f3f​​​
[2]​​​https://www.php.cn/link/67b4e63655366f054314061dadd539a0​​​
[3] ​​​https://www.php.cn/link/44590aa922914066f965ae67be0222d2​​

The above is the detailed content of Verification codes can’t stop robots! Google AI can accurately identify blurry text, while GPT-4 pretends to be blind and asks for help. For more information, please follow other related articles on the PHP Chinese website!