Microsoft's super powerful small model sparks heated discussion: exploring the huge role of textbook-level data

As large models set off a new round of AI craze, people began to think: What is the source of the powerful capabilities of large models?

Currently, large models have been driven by the ever-increasing amount of "big data". "Big Model Big Data" seems to have become the standard paradigm for building models. However, as the model size and data volume continue to grow, the demand for computing power will expand rapidly. Some researchers are trying to explore new ideas. Rewritten content: Currently, large-scale models have been driven by ever-increasing amounts of “big data.” "Large Model Big Data" seems to have become the standard paradigm for building models. However, as the model size and data volume continue to grow, the computing power requirements will expand rapidly. Some researchers are trying to explore new ideas

Microsoft released a paper called "Just Textbooks" in June, using a data set of only 7B markers to A model containing 1.3B parameters, called phi-1, was trained. Despite having datasets and model sizes that are orders of magnitude smaller than competitors, phi-1 achieved a first-time pass rate of 50.6% in the HumanEval test and 55.5% in the MBPP test

phi-1 proves that high-quality "small data" can give the model good performance. Recently, Microsoft published a paper "Textbooks Are All You Need II: phi-1.5 technical report" to further study the potential of high-quality "small data".

Paper address: https://arxiv.org/abs/2309.05463

Model Introduction

Architecture

The research team used phi-1 research methods and focused their research on natural language For common sense reasoning tasks, a Transformer architecture language model phi-1.5 with 1.3B parameters was developed. The architecture of phi-1.5 is exactly the same as phi-1, with 24 layers, 32 heads, each head has a dimension of 64, and uses a rotation embedding with a rotation dimension of 32, and a context length of 2048

In addition, this study also uses flash-attention for training acceleration and codegen-mono's tokenizer.

The content that needs to be rewritten is: training data

phi The content that needs to be rewritten for -1.5 is: the training data is composed of the training data (7B tokens) and the newly created "textbook quality" data (about 20B tokens) for phi-1. Among them, the newly created "textbook quality" data is designed to allow the model to master common sense reasoning, and the research team carefully selected 20K topics to generate new data.

It is worth noting that in order to explore the importance of network data (commonly used in LLM), this study also constructed two models: phi-1.5-web-only and phi-1.5-web .

The research team stated: Creating a powerful and comprehensive dataset requires not only raw computing power, but also complex iterations, effective topic selection, and a deep understanding of knowledge. These elements can ensure the quality and diversity of data.

Experimental results

This study evaluated language understanding tasks, using multiple data sets, including PIQA, Hellaswag, OpenbookQA, SQUAD and MMLU. The evaluation results are shown in Table 3. The performance of phi-1.5 is comparable to that of a model 5 times larger.

on the common sense reasoning benchmark. The test results are shown in the table below:

In more complex reasoning tasks, such as elementary school mathematics and basic coding tasks, phi-1.5 outperforms Most of the LLM

research team believes that phi-1.5 once again proves the power of high-quality "small data".

Questions and Discussions

Perhaps because the concept of "big model and big data" is too deeply rooted in the hearts of the people, this research has been criticized Some researchers in the machine learning community are skeptical, and some even suspect that phi-1.5 was trained directly on the test benchmark data set.

Netizen Susan Zhang conducted a series of verifications and pointed out: "phi-1.5 can give completely correct answers to the original problem in the GSM8K data set. answer, but as long as the format is slightly modified (such as line breaks), phi-1.5 will not answer."

Also modify the data in the question, phi-1.5 will cause "illusion" in the process of answering the question. For example, in a food ordering problem, if only the "price of pizza" is modified, the phi-1.5 answer will be wrong.

##And, phi-1.5 seems to "remember" Final answer, even if the answer is already wrong even if the data is modified.

In this regard, Ronan Eldan, an author of the paper, quickly responded and explained and refuted the problems that appeared in the above-mentioned netizen test:

#But the netizen once again stated his point of view: The test shows that the answer to phi-1.5 is very "fragile" to the format of the prompt, and is harmful to the author's Response to the question:

Li Yuanzhi, the first author of the paper, responded: "Although phi-1.5 is indeed inferior to GPT-4 in terms of robustness, but "Fragile" is not an accurate term. In fact, for any model, pass@k accuracy will be much higher than pass@1 (so the correctness of the model is accidental)

After seeing these questions and discussions, netizens said: “The easiest way to respond is to make the synthetic data set public. ”

What do you think of this?

The above is the detailed content of Microsoft's super powerful small model sparks heated discussion: exploring the huge role of textbook-level data. For more information, please follow other related articles on the PHP Chinese website!