Latest PNAS research: 81% problem solving rate, neural network Codex opens the door to the world of advanced mathematics

Recently, a new study was published in PNAS, which once again refreshed the capabilities of neural networks. This time the neural network was used to solve advanced mathematics problems, and they were difficult mathematics problems in the MIT mathematics course!

In this new study, the research team proved that OpenAI’s Codex model can perform program synthesis to solve large-scale mathematical problems, and automatically solve 81% of the data set through small sample learning mathematics course problems, and Codex achieved human-level performance on these tasks.

##Original link: https://www.pnas.org/doi/10.1073/pnas.2123433119

The emergence of this research subverts the common consensus that neural networks cannot solve advanced mathematics problems. The research team pointed out that the reason why Codex can achieve such capabilities is precisely because the team has made a major innovation. Those unsuccessful studies in the past only used text-based pre-training, and the Codex neural network that appeared this time not only Pre-training is done based on text, and the code is also fine-tuned.

The question data set studied was selected from six mathematics courses at MIT and one mathematics course at Columbia University. 25 questions were randomly selected from seven courses: MIT's Univariate Micron Integrals, multivariable calculus, differential equations, introduction to probability and statistics, linear algebra and mathematics for computer science and COMS3251 Computational Linear Algebra from Columbia University.

At the same time, the research team used MATH, the latest advanced mathematics problem benchmark for evaluating mathematical reasoning, to test the capabilities of OpenAI Codex. MATH was drawn from 6 major mathematics sections: Junior Algebra , 15 problems each from Algebra, Counting and Probability, Intermediate Algebra, Number Theory, and Precalculus.

Caption: Course question data set and MATH benchmark used in the study

Research shows that Codex solved 265 problems in the problem data set and MATH data set, of which 213 were solved automatically.

1 Where is the innovation

After the release of Transformer, language models based on Transformer have been used in various natural language processing (NLP) tasks, including It has achieved great success in zero-shot and few-shot language tasks. But because Transformer is only pre-trained on text, these models are basically unable to solve mathematical problems. GPT-3 is a typical example.

Later, through few-shot learning and Chain-of-thought (CoT) prompts, GPT-3’s mathematical reasoning capabilities were improved; however, , without code, even with small sample learning and CoT hints, GPT-3 is still powerless on college-level math problems and MATH benchmarks.

Past research on solving mathematical problems may have achieved certain results at a relatively simple mathematical level. For example, techniques that validate or predict expression trees based on collaborative training outputs, such as MAWPS and Math23k, can solve elementary school-level math problems with over 81% accuracy, but they cannot solve high school, Olympiad math, or college-level math problems. course. Co-training combined with graph neural networks (GNN) to predict arithmetic expression trees enables solving university-level problems in machine learning with up to 95% accuracy. But this work was also limited to numerical answers, and produced overfitting and could not be generalized to other courses.

One of the biggest innovations of this work is that not only the Transformer model such as Codex is pre-trained on the text, but also on the code fine-tuned so that it can generate programs that solve mathematical problems on a large scale.

The research team randomly selected question samples from the data set that did not require input images or proofs for testing. Among them, a language model pretrained only on text (GPT-3 text-davinci-002) automatically solved only 18% of the course problems and 25.5% of the MATH benchmark problems.

In contrast, a program synthesized using zero-shot learning and a neural network pretrained on text and fine-tuned on code (OpenAI Codex code-davinci-002) can automatically Solved 71% of course questions and 72.2% of MATH benchmark questions.

Using the same neural network Codex plus few-shot learning can automatically solve 81% of the problems in the course and 81.1% of the problems in the MATH benchmark test. However, 19% of course questions and 18.9% of MATH benchmark questions that the remaining models could not automatically solve were finally solved through manual prompts.

The addition of the small sample learning method is the second major innovation of this research. As can be seen from the above figure, when zero-shot learning cannot answer the question, (question, code) will be used to perform small-shot learning on (pair):

1) Use OpenAI The text-similarity-babbage-001 embedding engine embeds all questions;

2) Uses the embedded cosine similarity to calculate the solved questions from its course that are most similar to the unsolved questions;

3) Take the most similar problems and their corresponding codes as examples of small sample problems.

Illustration: Comparison of automatic problem-solving rates of 4 methods

The above picture shows the comparison of the automatic problem solving rates of Codex's zero-sample learning, small-sample learning, and GPT-3's zero-sample learning and small-sample learning. It can be seen from the figure that the small sample learning Codex represented by the orange bar has excellent performance in automatic problem solving rate, and its performance in basically every mathematical field is stronger than the other three methods.

The third major innovation of this research is to provide a pipeline for solving mathematical problems and explaining why they are solved as they are. The figure below shows MIT 5 The execution flow of pipelines in a mathematics course.

Taking the 18.01 single variable calculus problem as an example, given a problem and the automatically generated prefix "Use SymPy", the Codex is prompted and output a program. Running the program produces an equation with the correct answer. The program then automatically prompts for the Codex again, resulting in a generated code explanation.

2 After Problem Solving

In addition to solving math problems and explaining answers, Codex is also used to generate new questions for each course.

To evaluate the level of questions generated, the team conducted a survey among MIT students who had taken these courses or courses at the same level, mainly to compare the quality of machine-generated questions and manually written questions. and difficulty.

In each of MIT's 6 courses, 5 hand-written questions and 5 model-generated questions are mixed and presented randomly. For each of the 60 questions, participating students were asked to answer 3 survey questions:

1) Do you think this question was human-written or machine-generated?

2) Do you think this question is appropriate or inappropriate for a specific course?

3 ) On a scale of 1 (easiest) and 5 (hardest), what would you rate the difficulty level of this problem?

In the returned questionnaires, the student survey results are summarized as follows:

• The difficulty of machine-generated and manually written questions is similar.
• Human-written questions are more appropriate for the course than machine-generated questions.
• Human-written answers are difficult to identify incorrectly, while machine-generated questions are considered by students to be either machine-generated or human-written.

#The machine-generated questions have become indistinguishable to students, indicating that Codex has reached human performance levels in generating new content.

However, the model also has problems that it cannot solve, such as if the question appears in an image or other non-text form, it cannot answer; questions with solutions that need to be proved, Or computationally intractable problems, such as factoring very large prime numbers, cannot be solved by this model. However, this last type of question should not appear in any mathematics coursework because even real students cannot answer it.

The above is the detailed content of Latest PNAS research: 81% problem solving rate, neural network Codex opens the door to the world of advanced mathematics. For more information, please follow other related articles on the PHP Chinese website!