search
HomeTechnology peripheralsAITo supplement mathematics for large models, submit the open source MathPile corpus with 9.5 billion tokens, which can also be used commercially

In the current development of intelligent dialogue models, powerful underlying models play a crucial role. The pre-training of these advanced models often relies on high-quality and diverse corpora, and how to build such a corpus has become a major challenge in the industry.

In the high-profile field of AI for Math, the relative scarcity of high-quality mathematical corpus limits the potential of generative artificial intelligence in mathematical applications.

#To address this challenge, the Generative Artificial Intelligence Laboratory of Shanghai Jiao Tong University launched “MathPile”. This is a high-quality, diverse pre-training corpus specifically targeted at the field of mathematics, which contains approximately 9.5 billion tokens and is designed to improve the capabilities of large models in mathematical reasoning.

In addition, the laboratory also launched the commercial version of MathPile - "MathPile_Commercial" to further broaden its application scope and commercial potential.

To supplement mathematics for large models, submit the open source MathPile corpus with 9.5 billion tokens, which can also be used commercially

  • Paper address: https://huggingface.co/papers/2312.17120

  • Project address: https://gair-nlp.github.io/MathPile/
  • Code address: https://github.com/GAIR-NLP/MathPile

Dataset address:

  • Research use: https://huggingface.co/datasets /GAIR/MathPile
  • Commercial version: https://huggingface.co/datasets/GAIR/MathPile_Commercial

Introduction

MathPile has the following characteristics:

To supplement mathematics for large models, submit the open source MathPile corpus with 9.5 billion tokens, which can also be used commercially

1. Mathematically centered: Different from the corpora that focused on general fields in the past, such as Pile, RedPajama, or the multilingual corpus ROOTS, etc., MathPile focuses on the field of mathematics. Although there are already some specialized mathematical corpora, they are either not open source (such as the corpus used by Google to train Minerva, OpenAI's MathMix), or are not rich and diverse enough (such as ProofPile and the recent OpenWebMath).

2. Diversity: MathPile has a wide range of data sources, such as open source mathematics textbooks, class notes, synthetic textbooks, and mathematics on arXiv Related papers, math-related entries on Wikipedia, lemma proofs and definitions on ProofWiki, high-quality math questions and answers on StackExchange, and math web pages from Common Crawl. The above content covers content suitable for primary and secondary schools, universities, graduate students, and mathematics competitions. MathPile covers 0.19B tokens of high-quality mathematics textbooks for the first time.

3. High Quality: The research team follows the concept of "less is more" during the collection process and firmly believes in data quality Better than quantity, even in the pre-training phase. From a data source of ~520B tokens (approximately 2.2TB), they went through a rigorous and complex set of pre-processing, pre-filtering, language identification, cleaning, filtering and deduplication steps to ensure the high quality of the corpus. It is worth mentioning that the MathMix used by OpenAI only has 1.5B tokens.

4. Data documentation: In order to increase transparency, the research team documented MathPile and provided a dataset sheet. During the data processing process, the research team also performed "quality annotation" on documents from the Web. For example, the score of language recognition and the ratio of symbols to words in the document allow researchers to further filter documents according to their own needs. They also performed downstream test set contamination detection on the corpus to eliminate samples from benchmark test sets such as MATH and MMLU-STEM. At the same time, the research team also discovered that there are also a large number of downstream test samples in OpenWebMath, which shows that extra care should be taken when producing pre-training corpus to avoid invalid downstream evaluation.

To supplement mathematics for large models, submit the open source MathPile corpus with 9.5 billion tokens, which can also be used commercially

MathPile’s data collection and processing process.

Data processing details

Today, as competition in the field of large models becomes increasingly fierce, many technology companies no longer disclose their data; of data sources, matching, not to mention detailed preprocessing details. On the contrary, MathPile summarizes a set of data processing methods suitable for the Math field based on previous explorations.

In the data cleaning and filtering part, the specific steps adopted by the research team are:

  • Detect lines containing "lorem ipsum". If "lorem ipsum" is replaced by less than 5 characters in the line, remove the line;
  • Detect lines containing "javescript" ” and contains a line with "enable", "disable" or "browser", and the number of characters in the line is less than 200 characters, then filter out the line;
  • Filter out less than 200 characters. 10 words and lines containing "Login", "sign-in", "read more...", or "items in cart";
  • Filter out the proportion of uppercase words More than 40% of the documents;
  • Filter out the documents whose lines ending with an ellipsis account for more than 30% of the entire document;
  • Filter out the non- Documents where the proportion of letter words exceeds 80%;
  • Filter out documents whose average English word character length is outside the range of (3, 10);
  • Filter out documents that do not contain at least two stop words (such as the, be, to, of, and, that, have, etc.);
  • Filter out ellipses and Documents with a word proportion exceeding 50%;
  • Filter out documents in which the line starting with a bullet accounts for more than 90%;
  • Filter out documents with lines starting with bullets Documents of less than 200 characters after excluding spaces and punctuation;
  • ...

More processing details can be found in the paper.

In addition, the research team also provided many data samples during the cleaning process. The picture below shows the nearly duplicate documents in Common Crawl detected by the MinHash LSH algorithm (shown in pink highlights).

To supplement mathematics for large models, submit the open source MathPile corpus with 9.5 billion tokens, which can also be used commercially

As shown in the figure below, the research team discovered problems from the MATH test set (as highlighted in yellow) during the data leakage detection process.

To supplement mathematics for large models, submit the open source MathPile corpus with 9.5 billion tokens, which can also be used commercially

Data set statistics and examples

The following table shows the various components of MathPile According to the statistical information, you can find arXiv papers. Textbooks usually have longer document lengths, while documents on wikis are relatively shorter.

To supplement mathematics for large models, submit the open source MathPile corpus with 9.5 billion tokens, which can also be used commercially

The picture below is a sample document of a textbook in the MathPile corpus. It can be seen that the document structure is relatively clear and the quality is high.

To supplement mathematics for large models, submit the open source MathPile corpus with 9.5 billion tokens, which can also be used commercially

Experimental results

The research team also disclosed some preliminary experimental results. They performed further pre-training based on the currently popular Mistral-7B model. Then, it was evaluated on some common mathematical reasoning benchmark data sets through a few-shot prompting method. The preliminary experimental data that have been obtained so far are as follows:

To supplement mathematics for large models, submit the open source MathPile corpus with 9.5 billion tokens, which can also be used commercially

These test benchmarks cover all levels of mathematical knowledge, including primary school mathematics (such as GSM8K, TAL-SCQ5K-EN and MMLU- Math), high school mathematics (such as MATH, SAT-Math, MMLU-Math, AQuA, and MathQA), and college mathematics (such as MMLU-Math). Preliminary experimental results announced by the research team show that by continuing to pre-train on textbooks and Wikipedia subsets in MathPile, the language model has achieved considerable improvements in mathematical reasoning capabilities at different difficulty levels.

The research team also emphasized that relevant experiments are still ongoing.

Conclusion

MathPile has received widespread attention since its release and has been reproduced by many parties. It is currently listed on the Huggingface Datasets trend list. The research team stated that they will continue to optimize and upgrade the data set to further improve data quality.

To supplement mathematics for large models, submit the open source MathPile corpus with 9.5 billion tokens, which can also be used commercially

MathPile is trending on Huggingface Datasets. To supplement mathematics for large models, submit the open source MathPile corpus with 9.5 billion tokens, which can also be used commercially
MathPile was forwarded by the well-known AI blogger AK, source: https://twitter.com/_akhaliq/status/1740571256234057798.

Currently, MathPile has been updated to the second version, aiming to contribute to the research and development of the open source community. At the same time, its commercial version of the data set has also been opened to the public.

The above is the detailed content of To supplement mathematics for large models, submit the open source MathPile corpus with 9.5 billion tokens, which can also be used commercially. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:机器之心. If there is any infringement, please contact admin@php.cn delete
DSA如何弯道超车NVIDIA GPU?DSA如何弯道超车NVIDIA GPU?Sep 20, 2023 pm 06:09 PM

你可能听过以下犀利的观点:1.跟着NVIDIA的技术路线,可能永远也追不上NVIDIA的脚步。2.DSA或许有机会追赶上NVIDIA,但目前的状况是DSA濒临消亡,看不到任何希望另一方面,我们都知道现在大模型正处于风口位置,业界很多人想做大模型芯片,也有很多人想投大模型芯片。但是,大模型芯片的设计关键在哪,大带宽大内存的重要性好像大家都知道,但做出来的芯片跟NVIDIA相比,又有何不同?带着问题,本文尝试给大家一点启发。纯粹以观点为主的文章往往显得形式主义,我们可以通过一个架构的例子来说明Sam

阿里云通义千问14B模型开源!性能超越Llama2等同等尺寸模型阿里云通义千问14B模型开源!性能超越Llama2等同等尺寸模型Sep 25, 2023 pm 10:25 PM

2021年9月25日,阿里云发布了开源项目通义千问140亿参数模型Qwen-14B以及其对话模型Qwen-14B-Chat,并且可以免费商用。Qwen-14B在多个权威评测中表现出色,超过了同等规模的模型,甚至有些指标接近Llama2-70B。此前,阿里云还开源了70亿参数模型Qwen-7B,仅一个多月的时间下载量就突破了100万,成为开源社区的热门项目Qwen-14B是一款支持多种语言的高性能开源模型,相比同类模型使用了更多的高质量数据,整体训练数据超过3万亿Token,使得模型具备更强大的推

ICCV 2023揭晓:ControlNet、SAM等热门论文斩获奖项ICCV 2023揭晓:ControlNet、SAM等热门论文斩获奖项Oct 04, 2023 pm 09:37 PM

在法国巴黎举行了国际计算机视觉大会ICCV(InternationalConferenceonComputerVision)本周开幕作为全球计算机视觉领域顶级的学术会议,ICCV每两年召开一次。ICCV的热度一直以来都与CVPR不相上下,屡创新高在今天的开幕式上,ICCV官方公布了今年的论文数据:本届ICCV共有8068篇投稿,其中有2160篇被接收,录用率为26.8%,略高于上一届ICCV2021的录用率25.9%在论文主题方面,官方也公布了相关数据:多视角和传感器的3D技术热度最高在今天的开

复旦大学团队发布中文智慧法律系统DISC-LawLLM,构建司法评测基准,开源30万微调数据复旦大学团队发布中文智慧法律系统DISC-LawLLM,构建司法评测基准,开源30万微调数据Sep 29, 2023 pm 01:17 PM

随着智慧司法的兴起,智能化方法驱动的智能法律系统有望惠及不同群体。例如,为法律专业人员减轻文书工作,为普通民众提供法律咨询服务,为法学学生提供学习和考试辅导。由于法律知识的独特性和司法任务的多样性,此前的智慧司法研究方面主要着眼于为特定任务设计自动化算法,难以满足对司法领域提供支撑性服务的需求,离应用落地有不小的距离。而大型语言模型(LLMs)在不同的传统任务上展示出强大的能力,为智能法律系统的进一步发展带来希望。近日,复旦大学数据智能与社会计算实验室(FudanDISC)发布大语言模型驱动的中

百度文心一言全面向全社会开放,率先迈出重要一步百度文心一言全面向全社会开放,率先迈出重要一步Aug 31, 2023 pm 01:33 PM

8月31日,文心一言首次向全社会全面开放。用户可以在应用商店下载“文心一言APP”或登录“文心一言官网”(https://yiyan.baidu.com)进行体验据报道,百度计划推出一系列经过全新重构的AI原生应用,以便让用户充分体验生成式AI的理解、生成、逻辑和记忆等四大核心能力今年3月16日,文心一言开启邀测。作为全球大厂中首个发布的生成式AI产品,文心一言的基础模型文心大模型早在2019年就在国内率先发布,近期升级的文心大模型3.5也持续在十余个国内外权威测评中位居第一。李彦宏表示,当文心

AI技术在蚂蚁集团保险业务中的应用:革新保险服务,带来全新体验AI技术在蚂蚁集团保险业务中的应用:革新保险服务,带来全新体验Sep 20, 2023 pm 10:45 PM

保险行业对于社会民生和国民经济的重要性不言而喻。作为风险管理工具,保险为人民群众提供保障和福利,推动经济的稳定和可持续发展。在新的时代背景下,保险行业面临着新的机遇和挑战,需要不断创新和转型,以适应社会需求的变化和经济结构的调整近年来,中国的保险科技蓬勃发展。通过创新的商业模式和先进的技术手段,积极推动保险行业实现数字化和智能化转型。保险科技的目标是提升保险服务的便利性、个性化和智能化水平,以前所未有的速度改变传统保险业的面貌。这一发展趋势为保险行业注入了新的活力,使保险产品更贴近人民群众的实际

致敬TempleOS,有开发者创建了启动Llama 2的操作系统,网友:8G内存老电脑就能跑致敬TempleOS,有开发者创建了启动Llama 2的操作系统,网友:8G内存老电脑就能跑Oct 07, 2023 pm 10:09 PM

不得不说,Llama2的「二创」项目越来越硬核、有趣了。自Meta发布开源大模型Llama2以来,围绕着该模型的「二创」项目便多了起来。此前7月,特斯拉前AI总监、重回OpenAI的AndrejKarpathy利用周末时间,做了一个关于Llama2的有趣项目llama2.c,让用户在PyTorch中训练一个babyLlama2模型,然后使用近500行纯C、无任何依赖性的文件进行推理。今天,在Karpathyllama2.c项目的基础上,又有开发者创建了一个启动Llama2的演示操作系统,以及一个

腾讯与中国宋庆龄基金会发布“AI编程第一课”,教育部等四部门联合推荐腾讯与中国宋庆龄基金会发布“AI编程第一课”,教育部等四部门联合推荐Sep 16, 2023 am 09:29 AM

腾讯与中国宋庆龄基金会合作,于9月1日发布了名为“AI编程第一课”的公益项目。该项目旨在为全国零基础的青少年提供AI和编程启蒙平台。只需在微信中搜索“腾讯AI编程第一课”,即可通过官方小程序免费体验该项目由北京师范大学任学术指导单位,邀请全球顶尖高校专家联合参研。“AI编程第一课”首批上线内容结合中国航天、未来交通两项国家重大科技议题,原创趣味探索故事,通过剧本式、“玩中学”的方式,让青少年在1小时的学习实践中认识A

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
R.E.P.O. Best Graphic Settings
2 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
R.E.P.O. How to Fix Audio if You Can't Hear Anyone
3 weeks agoBy尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SAP NetWeaver Server Adapter for Eclipse

SAP NetWeaver Server Adapter for Eclipse

Integrate Eclipse with SAP NetWeaver application server.

mPDF

mPDF

mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

PhpStorm Mac version

PhpStorm Mac version

The latest (2018.2.1) professional PHP integrated development tool

Dreamweaver Mac version

Dreamweaver Mac version

Visual web development tools