


GPT-4 performs poorly in graph inference? Even after 'releasing water', the accuracy rate is only 33%
GPT-4’s graphical reasoning ability is less than half that of humans?
A study by the Santa Fe Research Institute in the United States shows that GPT-4’s accuracy in graphical reasoning questions is only 33%.
GPT-4v has multi-modal capabilities, but its performance is relatively poor. It can only answer 25% of the questions correctly
△Dotted line indicates Average performance of 16 tasks
As soon as the results of this experiment were released, they immediately caused extensive discussion on YC
Some netizens who supported this result said that GPT is indeed good at processing abstract graphics. Poor performance, and it is more difficult to understand concepts such as "position" and "rotation"
However, some netizens expressed doubts about this conclusion, and their views can be simply summarized as :
Although this view cannot be said to be wrong, it cannot be completely convincing
As for the specific reasons, we Read on.
The accuracy of GPT-4 is only 33%
In order to evaluate the performance of humans and GPT-4 on these graphics problems, the researchers used the ConceptARC data set launched in May this year
ConceptARC includes a total of 16 subcategories of graphical reasoning questions, with 30 questions in each category, for a total of 480 questions.
These 16 subcategories include positional relationships, shapes, operations, comparisons, etc.
Specifically, these questions are composed of Made up of pixel blocks. Humans and GPT need to find patterns based on given examples and analyze the results of images processed in the same way
The author specifically shows examples of these 16 subcategories in the paper, one for each category.
The results showed that the average accuracy rate of 451 human subjects was no less than 83% in each sub-item. Taking the average of 16 tasks, it reaches 91%.
In the case where a question can be tried three times (if it is correct once, it is correct), the highest accuracy of GPT-4 (single sample) does not exceed 60%, and the average is only 33%
Earlier, the author of ConceptARC Benchmark involved in this experiment also conducted a similar experiment, but in GPT-4 it was zero sample Test, the average accuracy rate of 16 tasks is only 19%.
GPT-4v is a multi-modal model, but its accuracy is low. On a small-scale ConceptARC data set consisting of 48 questions, the accuracy rates of zero-sample testing and single-sample testing were only 25% and 23% respectively.
And the researchers After further analyzing the wrong answers, it was found that some human errors seemed likely to be caused by "carelessness", while GPT did not understand the rules in the questions at all.
Most netizens have no doubts about these data, but what has caused this experiment to be questioned is the group of subjects recruited and the input method provided to GPT
The method of subject selection has been questioned
Initially, the researchers recruited subjects on an Amazon crowdsourcing platform.
The researcher extracted some simple questions from the data set as an introductory test. The subjectsneed to answer at least two of the three random questions correctly before entering the formal test .
The results found by the researchers show that some people only take the entrance test for the purpose of greed for money, but do not complete the questions as required
As a last resort, the researcherswill The threshold for participating in the test has been raised to the point where no less than 2,000 tasks have been completed on the platform, and the pass rate must reach 99%. However, although the author uses the pass rate to screen people, in terms of specific abilities, in addition to requiring subjects to speak English, there are "no special requirements" for
other professional abilities such as graphics . To achieve data diversity, the researchers moved recruitment efforts to another crowdsourcing platform later in the experiment. In the end, a total of 415 subjects participated in this experiment
Despite this, some people still questioned that the samples in the experiment were "
not random enough".
Some netizens pointed out that on the Amazon crowdsourcing platform used by researchers to recruit subjects,
there were large models posing as humans .
The operation of the multi-modal version of GPT is relatively simple. You only need to directly input the image and use the corresponding prompt word
In the zero-sample test, just delete the corresponding EXAMPLE part
But for the plain text version of GPT-4 (0613) without multi-modality, you need to convert the image into grid points ,
Use numbers instead of colors.
Some people disagree with this operation:
After converting the image into a digital matrix, the concept completely changes
, even humans, looking at the "graphics" represented by numbers, may not be able to understand
One More ThingCoincidentally , Joy Hsu, a Chinese doctoral student at Stanford University, also tested the graph understanding ability of GPT-4v on a geometric data set. Last year, a data set was released to test the understanding of Euclidean geometry by large models. After GPT-4v was opened, Hsu used the data set to test it again.
The result was that GPT-4v seemed to understand graphics "completely differently from humans."
In terms of data, GPT-4v is significantly inferior to humans in answering these geometric questionsPaper Address:
[1]https://arxiv.org/abs/2305.07141[2]https://arxiv.org/abs/2311.09247
The above is the detailed content of GPT-4 performs poorly in graph inference? Even after 'releasing water', the accuracy rate is only 33%. For more information, please follow other related articles on the PHP Chinese website!
![[Ghibli-style images with AI] Introducing how to create free images with ChatGPT and copyright](https://img.php.cn/upload/article/001/242/473/174707263295098.jpg?x-oss-process=image/resize,p_40)
The latest model GPT-4o released by OpenAI not only can generate text, but also has image generation functions, which has attracted widespread attention. The most eye-catching feature is the generation of "Ghibli-style illustrations". Simply upload the photo to ChatGPT and give simple instructions to generate a dreamy image like a work in Studio Ghibli. This article will explain in detail the actual operation process, the effect experience, as well as the errors and copyright issues that need to be paid attention to. For details of the latest model "o3" released by OpenAI, please click here⬇️ Detailed explanation of OpenAI o3 (ChatGPT o3): Features, pricing system and o4-mini introduction Please click here for the English version of Ghibli-style article⬇️ Create Ji with ChatGPT

As a new communication method, the use and introduction of ChatGPT in local governments is attracting attention. While this trend is progressing in a wide range of areas, some local governments have declined to use ChatGPT. In this article, we will introduce examples of ChatGPT implementation in local governments. We will explore how we are achieving quality and efficiency improvements in local government services through a variety of reform examples, including supporting document creation and dialogue with citizens. Not only local government officials who aim to reduce staff workload and improve convenience for citizens, but also all interested in advanced use cases.

Have you heard of a framework called the "Fukatsu Prompt System"? Language models such as ChatGPT are extremely excellent, but appropriate prompts are essential to maximize their potential. Fukatsu prompts are one of the most popular prompt techniques designed to improve output accuracy. This article explains the principles and characteristics of Fukatsu-style prompts, including specific usage methods and examples. Furthermore, we have introduced other well-known prompt templates and useful techniques for prompt design, so based on these, we will introduce C.

ChatGPT Search: Get the latest information efficiently with an innovative AI search engine! In this article, we will thoroughly explain the new ChatGPT feature "ChatGPT Search," provided by OpenAI. Let's take a closer look at the features, usage, and how this tool can help you improve your information collection efficiency with reliable answers based on real-time web information and intuitive ease of use. ChatGPT Search provides a conversational interactive search experience that answers user questions in a comfortable, hidden environment that hides advertisements

In a modern society with information explosion, it is not easy to create compelling articles. How to use creativity to write articles that attract readers within a limited time and energy requires superb skills and rich experience. At this time, as a revolutionary writing aid, ChatGPT attracted much attention. ChatGPT uses huge data to train language generation models to generate natural, smooth and refined articles. This article will introduce how to effectively use ChatGPT and efficiently create high-quality articles. We will gradually explain the writing process of using ChatGPT, and combine specific cases to elaborate on its advantages and disadvantages, applicable scenarios, and safe use precautions. ChatGPT will be a writer to overcome various obstacles,

An efficient guide to creating charts using AI Visual materials are essential to effectively conveying information, but creating it takes a lot of time and effort. However, the chart creation process is changing dramatically due to the rise of AI technologies such as ChatGPT and DALL-E 3. This article provides detailed explanations on efficient and attractive diagram creation methods using these cutting-edge tools. It covers everything from ideas to completion, and includes a wealth of information useful for creating diagrams, from specific steps, tips, plugins and APIs that can be used, and how to use the image generation AI "DALL-E 3."

Unlock ChatGPT Plus: Fees, Payment Methods and Upgrade Guide ChatGPT, a world-renowned generative AI, has been widely used in daily life and business fields. Although ChatGPT is basically free, the paid version of ChatGPT Plus provides a variety of value-added services, such as plug-ins, image recognition, etc., which significantly improves work efficiency. This article will explain in detail the charging standards, payment methods and upgrade processes of ChatGPT Plus. For details of OpenAI's latest image generation technology "GPT-4o image generation" please click: Detailed explanation of GPT-4o image generation: usage methods, prompt word examples, commercial applications and differences from other AIs Table of contents ChatGPT Plus Fees Ch

How to use ChatGPT to streamline your design work and increase creativity This article will explain in detail how to create a design using ChatGPT. We will introduce examples of using ChatGPT in various design fields, such as ideas, text generation, and web design. We will also introduce points that will help you improve the efficiency and quality of a variety of creative work, such as graphic design, illustration, and logo design. Please take a look at how AI can greatly expand your design possibilities. table of contents ChatGPT: A powerful tool for design creation


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Dreamweaver Mac version
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

EditPlus Chinese cracked version
Small size, syntax highlighting, does not support code prompt function

MinGW - Minimalist GNU for Windows
This project is in the process of being migrated to osdn.net/projects/mingw, you can continue to follow us there. MinGW: A native Windows port of the GNU Compiler Collection (GCC), freely distributable import libraries and header files for building native Windows applications; includes extensions to the MSVC runtime to support C99 functionality. All MinGW software can run on 64-bit Windows platforms.

SecLists
SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.
