Reading + AI
LibrarySign in
Open the original paper on arXiv (new tab)

Measuring Mathematical Problem Solving With the MATH Dataset

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, Jacob Steinhardt

arXiv:2103.03874v2 · This workspace stays on this version. Check arXiv for newer versions

Affiliations and author notes
  • UC Berkeley
  • UChicago
Some source details need review 1 to check

Original formulas, table values and figures are preserved. Comparison can miss differences; use the original PDF to check details.

Reading view checked: 178 math expressions, 0 rendering failures; 17 of 17 images available.

Compared 100 of 100 source passages against 22 of 22 PDF pages.

HTML S3_T2 says “5.4 +0\%5”; PDF page 5 shows “5.4 +0%” in the table image, contradicting the HTML. Check PDF page 5

17 of 17 figure, table and graphic images captured. Open original PDF

Abstract

Many intellectual endeavors require mathematical problem solving, but this skill remains beyond the capabilities of computers. To measure this ability in machine learning models, we introduce MATH, a new dataset of 12, ⁣50012,\!500 challenging competition mathematics problems. Each problem in MATH has a full step-by-step solution which can be used to teach models to generate answer derivations and explanations. To facilitate future research and increase accuracy on MATH, we also contribute a large auxiliary pretraining dataset which helps teach models the fundamentals of mathematics. Even though we are able to increase accuracy on MATH, our results show that accuracy remains relatively low, even with enormous Transformer models. Moreover, we find that simply increasing budgets and model parameter counts will be impractical for achieving strong mathematical reasoning if scaling trends continue. While scaling Transformers is automatically solving most other text-based tasks, scaling is not currently solving MATH. To have more traction on mathematical problem solving we will likely need new algorithmic advancements from the broader research community.

Mathematics is a highly effective tool in many intellectual endeavors. It enables us to count and quantify objects, and it can be relied upon because it is consistent and based on logic. Mathematics pervades the sciences and can be used to model planetary orbits, atomic motion, signal frequencies, and much more. These phenomena can be encoded with mathematics precisely and concisely. This has even led some to describe mathematics as being “unreasonably effective” (Wigner, 1960). These observations speak to the broad reach and domain-generality of mathematics.

In machine learning, mathematics is a valuable testbed for problem-solving ability: the ability to analyze a problem, pick out good heuristics from a large set of possibilities, and chain them together to produce an answer. This contrasts with plug-and-chug calculations, a skill which ML models can already exhibit (Henighan et al., 2020). Visual or linguistic reasoning may involve limited problem-solving ability for tasks such as image classification, but unlike math this is not the focus of these domains.

To measure the problem-solving ability of machine learning models, we introduce the MATH dataset, which consists of 12,50012,500 problems from high school math competitions. Given a problem from MATH, machine learning models generate a sequence, such as $\frac{2}{3}$, that encodes the final answer. These answers are unique after normalization, allowing MATH to be scored with exact match rather than with heuristic metrics such as BLEU. In addition, MATH problems are tagged by difficulty from 1 to 5, and span seven subjects including geometry, where diagrams can be specified in text with the Asymptote language. This enables a fine-grained assessment of mathematical problem-solving ability across difficulties and subjects. Finally, problems come with full step-by-step solutions. By training on these, models can learn to generate their own step-by-step solutions, which can facilitate learning and make model outputs more interpretable.

Figure 1
Figure 1: Previous work is based on formal theorem provers or straightforward plug-and-chug problems. Our dataset, MATH, has competition mathematics problems with step-by-step solutions written in LaTeX and natural language. Models are tasked with generating tokens to construct the final (boxed) answer.

The MATH dataset is challenging: large language models achieved accuracies ranging from 3.0%3.0\% to 6.9%6.9\%. Despite these low accuracies, models clearly possess some mathematical knowledge: they achieve up to 15%15\% accuracy on the easiest difficulty level, and they are able to generate step-by-step solutions that are coherent and on-topic even when incorrect. We also evaluated humans on MATH, and found that a computer science PhD student who does not especially like mathematics attained approximately 40%40\% on MATH, while a three-time IMO gold medalist attained 90%90\%, indicating that MATH can be challenging for humans as well.

The presence of step-by-step solutions allows models to utilize “scratch space”: rather than having to generate a final answer immediately, models can first generate solutions that may contain intermediate computations. Interestingly, we found that having models generate step-by-step solutions before producing an answer actually decreased accuracy relative to immediately outputting a final answer without generating solutions, indicating the solutions are currently not useful for models at test time. In contrast, having models train on solutions increases relative accuracy by 10%10\% compared to training on the questions and answers directly. We also find that models do better with hints in the form of partial solutions. Our results show that models can make use of actual step-by-step solutions provided to them in various ways, but that they are still unable to effectively use their own generated solutions. Bridging this gap poses an interesting direction for further research.

While MATH covers advanced problem-solving techniques, models may first need to be trained thoroughly on the fundamentals of mathematics. To address this, we create the first large-scale mathematics pretraining dataset with hundreds of thousands of step-by-step solutions in natural language and LaTeX. We call this dataset the Auxiliary Mathematics Problems and Solutions (AMPS) pretraining corpus, which consists of Khan Academy and Mathematica data. AMPS has over 100,000100,000 Khan Academy problems with step-by-step solutions in LaTeX; these exercises are used to teach human students concepts ranging from basic addition to Stokes’ Theorem. It also contains over 55 million problems generated using Mathematica scripts, based on 100100 hand-designed modules covering topics such as conic sections, div grad and curl, KL divergence, eigenvalues, polyhedra, and Diophantine equations. In total AMPS contains 23GB of problems and solutions. Pretraining on AMPS enables a 0.10.1 billion parameter model to perform comparably to a fine-tuned model that is 130×130\times larger.

Altogether, while large Transformer models (Vaswani et al., 2017) make some progress on the MATH dataset, such as by AMPS pretraining or by training with step-by-step solutions, accuracy nonetheless remains relatively low. While enormous Transformers pretrained on massive datasets can now solve most existing text-based tasks, this low accuracy indicates that our MATH dataset is distinctly harder. Accuracy also increases only modestly with model size: assuming a log-linear scaling trend, models would need around 103510^{35} parameters to achieve 40%40\% accuracy on MATH, which is impractical. Instead, to make large strides on the MATH dataset with a practical amount of resources, we will need new algorithmic advancements from the broader research community.

Neural Theorem Provers. Much of the existing work on machine learning models for mathematical reasoning relies on automated theorem proving benchmarks. Huang et al. (2019) use the Coq theorem proving environment to create a machine learning benchmark with 1,6021,602 theorems and lemmas. Bansal et al. (2019) introduce the HOList benchmark for automated theorem proving, which uses a formal language to enable automatic evaluation. Rather than use HOList, Polu and Sutskever (2020) use the Metamath formalization language for automated theorem proving with promising results. We show an example of Metamath in Figure 1. These benchmarks can be approached with seq2seq (Sutskever et al., 2014) Transformers which have traction on the problem (Polu and Sutskever, 2020; Rabe et al., 2020; Li et al., 2020).

Figure 2
Figure 2: Compared to existing proof and plug-and-chug tasks, our mathematical problem solving task is considerably more challenging. HOList results are from Wu et al. (2021). HOLStep results are from Crouse et al. (2019). DeepMind Math accuracy is the median IID accuracy from Henighan et al. (2020). Symbolic Integration accuracy is from Lample and Charton (2020).

Rather than prove theorems with standard pretrained Transformers, McAllester (2020) proposes that the community create theorem provers that bootstrap their mathematical capabilities through open-ended self-improvement. For bootstrapping to be feasible, models will also need to understand mathematics as humans write it, as manually converting advanced mathematics to a proof generation language is extremely time-consuming. This is why Szegedy (2020) argues that working on formal theorem provers alone will be an impractical path towards world-class mathematical reasoning. We address Szegedy (2020)’s concern by creating a dataset to test understanding of mathematics written in natural language and commonplace mathematical notation. This also means that the answers in our dataset can be assessed without the need for a cumbersome theorem proving environment, which is another advantage of our evaluation framework.

Neural Calculators. Recent work shows that Transformers can sometimes perform laborious calculations around as well as calculators and computer algebra systems. Lample and Charton (2020) use Transformers to solve algorithmically generated symbolic integration problems and achieve greater than 95% accuracy. Amini et al. (2019); Ling et al. (2017) introduce plug-and-chug multiple choice mathematics problems and focus on sequence-to-program generation. Saxton et al. (2019) introduce the DeepMind Mathematics dataset, which consists of algorithmically generated plug-and-chug problems such as addition, list sorting, and function evaluation, as shown in Figure 1. Recently, Henighan et al. (2020) show that, excluding problems with astronomically large numbers, the vast majority of the problems in the DeepMind Mathematics dataset can be straightforwardly solved with large Transformers.

Benchmarks for Enormous Transformers. There are few existing natural language benchmarks left to solve, as tasks that aggregate multiple subtasks such as SuperGLUE (Wang et al., 2019) are solved by simply training enormous Transformers (He et al., 2020). Kaplan et al. (2020); Henighan et al. (2020) show that the performance of Transformers predictably increases with an increase in model size and dataset size, raising the question of whether natural language processing can be solved by simply increasing compute and funding. Additionally, Chen et al. (2021); Austin et al. (2021) show that code generation models scale reliably across several orders of magnitude, and, should scaling continue, Chen et al. (2021)’s HumanEval code generation dataset should be solved in a few orders of magnitude. In the Supplementary Materials, we even find that large GPT-3 models can perform remarkably well on a sequence completion test similar to an IQ test, the C-Test (Hernández-Orallo, 1998; Legg and Hutter, 2007). Even difficult logical understanding tasks such as LogiQA (Liu et al., 2020) will soon be straightforwardly solved by enormous Transformers should trends continue, which we also show in the Supplementary Materials. Hendrycks et al. (2021) create a multiple-choice benchmark covering 5757 subjects. However, unlike our benchmark, which is a text generation task with 12,50012,500 mathematical reasoning questions, their benchmark is a multiple choice task that includes only a few hundred questions about mathematics. In contrast to these benchmarks, we find that our MATH benchmark is unusually challenging for current models and, if trends continue, simply using bigger versions of today’s Transformers will not solve our task in the foreseeable future.

Table 1
AlgebraConic sections, polynomial GCD, De Moivre’s theorem, function inverses, …
CalculusArclength, Jacobian, Laplacian, divergence, curl, gradients, integrals, …
StatisticsExpectation, geometric mean, harmonic mean, KL divergence, variance, …
GeometryTriangle area, triangle inradius, polygon angles, polyhedron diameter, …
Linear AlgebraCharacteristic polynomials, eigenvalues, reduced row echelon form, …
Number TheoryModular inverse, Euler’s totient function, Chinese remainder theorem, …
Table 1: A subset of the topics covered by our 100100 hand-designed Mathematica scripts, which is part of our Auxiliary Mathematics Problems and Solutions (AMPS) pretraining dataset. Of these scripts, 3737 also generate step-by-step solutions. We generated around 50, ⁣00050,\!000 exercises with each Mathematica script, or around 55 million problems.
View original tableOriginal Table 1

In this section, we introduce two new datasets, one for benchmarking mathematical problem-solving ability (MATH) and one for pretraining (AMPS).

The MATH dataset consists of problems from mathematics competitions including the AMC 10, AMC 12, AIME, and more. Many of these problems can be collected from aops.com/community/c3158_usa_contests. These competitions span decades and assess the mathematical problem-solving ability of the best young mathematical talent in the United States. Unlike most prior work, most problems in MATH cannot be solved with a straightforward application of standard K-12 mathematics tools. Instead, humans often solve such problem by applying problem solving techniques and “heuristics” (Pólya, 1945).

The Mathematics Aptitude Test of Heuristics dataset, abbreviated MATH, has 12, ⁣50012,\!500 problems (7, ⁣5007,\!500 training and 5, ⁣0005,\!000 test). With this many training problems, models can learn many useful heuristics for problem solving. Each problem has a step-by-step solution and a final boxed answer. Example problems with step-by-step solutions are shown in Figure 1.

Categorizing Problems. Problems span various subjects and difficulties. The seven subjects are Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus. While subjects like Prealgebra are generally easier than Precalculus, within a subject problems can take on different difficulty levels. We encode a problem’s difficulty level from ‘11’ to ‘55,’ following AoPS. A subject’s easiest problems for humans are assigned a difficulty level of ‘11,’ and a subject’s hardest problems are assigned a difficulty level of ‘55.’ Concretely, the first few problems of an AMC 8 exam are often level 11, while AIME problems are level 55. This allows us to assess performance across both different subjects and different levels of difficulty.

Formatting. Problems and solutions are consistently formatted using LaTeX and the Asymptote vector graphics language. Our usage of LaTeX allows us to flexibly encode mathematical problems while avoiding unusual symbols or cumbersome formal languages. Meanwhile, mathematical figures are encoded in the Asymptote language rather than as raster images. This enables pure language models to process figures, diagrams, and graphics, making it possible to assess these models on subjects such as geometry for the first time.

To assess models using exact match, we force the final boxed answers to follow consistent formatting rules. Specifically, probabilities are expressed as simplified fractions. Moreover, matrix entry fractions are encoded with x/y, while all other fractions are consistently encoded with the \frac{x}{y} command. Coefficients are encoded without a multiplication symbol (e.g. 5x not 5*x). Expressions with multiple variables are entered in alphabetical order; polynomials are expressed in decreasing degree order. Different fraction encodings equivalent, such as \frac{x}{y} and \dfrac{x}{y} and x/y. Different parenthesis encodings, such as \left( and (, are treated as equivalent.

We also allow units to be included or omitted from an answer, we ignore spaces, and we treat common equivalent ways of expressing the same number (e.g., 0.50.5 and 1/21/2, or 0.10.1 and .1.1) as the same. When the answer is a factorized polynomial, we permit different orderings of the factors, so that 4(x+1)(x−1)4(x+1)(x-1) is equivalent to 4(x−1)(x+1)4(x-1)(x+1), and so on. These rules cover nearly all ways that different generated or actual solutions can be equivalent in practice.

Automatically Assessing Generated Answers. Due to design choices in MATH, we can assess the answers generated by a model automatically, even though the space of model outputs is combinatorially large. Automatic assessment starts by determining the beginning and end of the answer. This is possible to do even if a model generates step-by-step solutions because the final answers in MATH are wrapped and delimited with the \boxed{} command. We can consequently evaluate a model’s output by parsing what is inside the \boxed{} command and comparing that with the ground truth answer, while accounting for the equivalent ways of formatting a string described above. Together, the box delimiter and formatting rules provide a unique answer in a well-defined location, which allows us to test for equivalence and use accuracy as our primary metric.

Human-Level Performance. To provide a rough but informative comparison to human-level performance, we randomly sampled 2020 problems from the MATH test set and gave them to humans. We artificially require that the participants have 1 hour to work on the problems and must perform calculations by hand. All participants are university students. One participant who does not like mathematics got 8/20=40%8/20=40\% correct. A participant ambivalent toward mathematics got 13/2013/20. Two participants who like mathematics got 14/2014/20 and 15/2015/20. A participant who got a perfect score on the AMC 10 exam and attended USAMO several times got 18/2018/20. A three-time IMO gold medalist got 18/20=90%18/20=90\%, though missed questions were exclusively due to small errors of arithmetic. Expert-level performance is theoretically 100%100\% given enough time. Even 40%40\% would accuracy for a machine learning model would be impressive but have ramifications for cheating on homework.

Table 2
ModelPrealgebraAlgebraNumber TheoryCounting & ProbabilityGeometryIntermediate AlgebraPrecalculusAverage
GPT-2 0.1B5.25.15.02.85.76.57.35.4 +0%+0\%5
GPT-2 0.3B6.76.65.53.86.96.07.16.2 +15%+15\%
GPT-2 0.7B6.96.15.55.18.25.87.76.4 +19%+19\%
GPT-2 1.5B8.36.24.85.48.76.18.86.9 +28%+28\%
GPT-3 13B*4.12.43.34.51.03.22.03.0 −44%-44\%
GPT-3 13B6.85.35.54.17.14.75.85.6 +4%+4\%5
GPT-3 175B*7.76.04.44.73.14.44.05.2 −4%-4\%5
Table 2: MATH accuracies across subjects. ‘*’ indicates that the model is a few-shot model. The character ‘B’ denotes the number of parameters in billions. The gray text indicates the relative improvement over the 0.1B baseline. All GPT-2 models pretrain on AMPS, and all values are percentages. GPT-3 models do not pretrain on AMPS due to API limits. Model accuracy is increasing very slowly, so much future research is needed.
View original tableOriginal Table 2

Since pretraining data can greatly influence performance (Hernandez et al., 2021; Gururangan et al., 2020) and since mathematics is a small fraction of online text, we introduce a large and diverse mathematics pretraining corpus. Our pretraining dataset, the Auxiliary Mathematics Problems and Solutions (AMPS) dataset, has problems and step-by-step solutions typeset in LaTeX. AMPS contains over 100, ⁣000100,\!000 problems pulled from Khan Academy and approximately 55 million problems generated from manually designed Mathematica scripts.

Khan Academy. The Khan Academy subset of AMPS has 693693 exercise types with over 100, ⁣000100,\!000 problems and full solutions. Problem types range from elementary mathematics (e.g. addition) to multivariable calculus (e.g. Stokes’ theorem), and are used to teach actual K-12 students. The exercises can be regenerated using code from github.com/Khan/khan-exercises/. We show the full list of problem types in the Supplementary Materials.

Mathematica. To make AMPS larger, we also contribute our own Mathematica scripts to generate approximately 50×50\times more problems than our Khan Academy dataset. With Mathematica, we designed 100100 scripts that test distinct mathematics concepts, 3737 of which include full step-by-step LaTeX solutions in addition to final answers. We generated around 50, ⁣00050,\!000 exercises from each of our scripts, or around 55 million problems in total. This results in over 2323 GB of mathematics problems, making it larger than the 1616 GB of natural language used to train BERT (Devlin et al., 2019).

Problems include various aspects of algebra, calculus, counting and statistics, geometry, linear algebra, and number theory (see Table 1 for a sampling of topics). Unlike prior approaches to algorithmically generating mathematics problems, we use Mathematica’s computer algebra system so that we can manipulate fractions, transcendental numbers, and analytic functions.

In this section, we perform experiments to investigate performance on the MATH dataset. We find that accuracy remains low even for the best models. Furthermore, unlike for most other text-based datasets, we find that accuracy is increasing very slowly with model size. If trends continue, then we will need algorithmic improvements, rather than just scale, to make substantial progress on MATH. Nevertheless, we show that making progress is also possible today. We find that pretraining on AMPS enables a small 0.1B parameter model to perform similarly to a large fine-tuned 13B parameter model.

We also experiment with using step-by-step solutions. We find that having models generate their own step-by-step solutions before producing an answer actually degrades accuracy. We qualitatively assess these generated solutions and find that while many steps remain illogical, they are often related to the question. Finally, we show that step-by-step solutions can still provide benefits today. We find that providing partial ground truth step-by-step solutions can improve performance, and that providing models with step-by-step solutions at training time also increases accuracy.

Figure 3
Figure 3: Problems, step-by-step solutions generated by our GPT-2 1.5B model, and ground truth solutions. Observe that models can provide the right answer yet generate a misleading and wrong explanation. The second generated solution demonstrates that models are capable of generating Asymptote commands to create figures and graphics.
Figure 4
Figure 4: Additional example problems, generated solutions, and ground truth solutions from our MATH dataset. The first problem’s generated solution has the right answer with a correct and simple explanation. The second problem is a combinatorics problem specified with a figure, which the model gets wrong.

Models and Hyperparameters. Because MATH answers must be generated, we use autoregressive language models, namely GPT-2 (Radford et al., 2016) and GPT-3 (Brown et al., 2020), which are decoder models pretrained on natural language text. Our GPT-2 models tokenizes numbers so that one digit is processed at a time (Henighan et al., 2020). T5’s (Raffel et al., 2020) tokenizer removes many LaTeX symbols, so after a broad hyperparameter sweep lasting two weeks, its performance was not competitive. We show results with the BART architecture in the Appendix.

Before fine-tuning on MATH, models pretrain on AMPS. We pretrain for one epoch, using AdamW (Loshchilov and Hutter, 2019), using a batch size of 128128, and using a weight decay of 0.050.05. We use the standard autoregressive language modeling objective. During pretraining, we upsample Khan Academy data by a factor of 55 and we downsample Mathematica by a factor of 22 to account for the large difference in dataset sizes.

During fine-tuning, models predict final answers and solutions. Concretely, if ⟨P⟩\langle P\rangle is the problem statement, we train with an equal mix of “⟨P⟩\langle P\rangle Final Answer: <Answer>” and “⟨P⟩\langle P\rangle Full Solution: <Step-by-Step Solution>” sequences. This makes it possible for the model to both generate full solutions and also to output just the final answer. For fine-tuning we use the same batch size and weight decay as in pretraining. Models are trained with 8 A100 GPUs, each requiring less than a day.

Unless otherwise specified, for GPT-2 we use the default HuggingFace (Wolf et al., 2020) generation parameters, except that we use beam search. Our beam search has a beam size of 2020 when only generating the final answer, and a beam size of 1010 when generating full step-by-step solutions. By default, we evaluate models by prompting them with “⟨P⟩\langle P\rangle Final Answer:” so that they directly generate the final answer to each problem, not the step-by-step solution.

We also evaluate GPT-3 with fine-tuning and also in a few-shot setting using the OpenAI API. We use the ‘Curie’ GPT-3 model which has approximately 1313 billion parameters, and the ‘Davinci’ model which has approximately 175175 billion parameters. When performing few-shot evaluation, we construct our prompt by prepending 88 problems with correct answers (but not step-by-step solutions due to space). Using temperature 00, models output up to 2020 tokens for the final answer. The OpenAI API also allows users to fine-tune models up to 13B parameters at the time of writing, but their API does not have the option to pretrain on datasets as large as AMPS.

AMPS Pretraining. As an ablation, we test how models with AMPS pretraining compare with models that were not pretrained on AMPS. Without pretraining on AMPS, a GPT-3 (13B) model fine-tuned on MATH attains 5.2%5.2\% accuracy. In contrast, a GPT-2 (0.1B) model both pretrained on AMPS and fine-tuned on MATH attains 5.4%5.4\%. Consequently AMPS increases accuracy about as much as a 130×130\times increase in parameters, demonstrating its value as a pretraining dataset.

We additionally tried pretraining on StackExchange, a real-world but less curated source of mathematics text. A GPT-2 (0.3B) model pretrained on both AMPS and questions and answers from Math StackExchange (∼ ⁣3\sim\!3 GB) had 6.0%6.0\% accuracy. This is actually less than the 6.2%6.2\% accuracy attained by pretraining on AMPS alone. Thus our dataset is more useful for pretraining even than diverse real-world mathematics data.

Model Size. While increasing model parameters often automatically solves many tasks (Brown et al., 2020), we find that MATH is unusually challenging for enormous Transformers. Table 2 shows that the average accuracy across subjects for the smallest model, GPT-2 with 0.1 billion parameters, is 5.4%5.4\%. Meanwhile, a GPT-2 model with 15×15\times the number of parameters attains 6.9%6.9\% accuracy, a 28%28\% relative improvement. This indicates that while having more parameters helps, absolute accuracy remains far from the ceiling and is only increasing slowly, quite unlike most other text-based tasks.

Problem Difficulty. We also analyze model accuracy while controlling for problem difficulty. Higher levels of difficulty correspond to lower accuracy, as expected. These results are visualized in the Supplementary Materials. The accuracy of GPT-2 (1.5B) is around 15%15\% for level 1 (easy) and around 4%4\% for level 5 (hard). Even our benchmark’s easiest problems are more challenging than previous benchmarks that focused on straightforward plug-and-chug problems.

Error Detection. To determine whether we can trust the answers from a model, we analyze model confidence to see whether confidence tends to be higher for correct answers. We define confidence as the average prediction probability of the tokens that make up a generated answer. GPT-2 (1.5B) is highly overconfident, with confidences that are often around 100%100\%. Moreover, there is substantial overlap between correct and incorrect answers. Following Hendrycks and Gimpel (2017), we computed the probability that a correct answer has higher confidence than an incorrect answer. To do this, we compute the Area Under the Receiver Operating Characteristic curve (AUROC). An AUROC of 100%100\% corresponds to being able to perfectly detect correct and incorrect answers, while 50%50\% corresponds to random chance. We find that with GPT-2 (1.5B), the AUROC is quite low at 68.8%68.8\%. This suggests there is substantial room for improvement in detecting model errors.

Figure 5
Figure 5: Models conditioned on most of a problem’s step-by-step solution can often understand the solution to predict the final answer. Not all solutions have an answer that is immediate from the preceding solution text, though many are. ‘99%’ of a solution is all the solution text before the final answer. This demonstrates that, even with substantial help, models are still struggling.

Scratch Space. Our MATH dataset and AMPS pretraining dataset provide full step-by-step solutions, an important and rare type of side information (Murty et al., 2020) that can in principle teach models how to derive answers and use scratch space. By training a language model on these solutions, we can have models generate full step-by-step solutions. This may be especially useful for difficult problems, for which outputting the correct answer after just a few forward passes may be insufficient. By allowing the model to use several steps of processing before outputting a final answer, the model could adaptively use computation and have higher performance, in addition to making its reasoning more interpretable.

We test this by prompting models with “⟨P⟩\langle P\rangle Full Solution:” to generate a full solution along with a final boxed answer, rather than the boxed answer alone. We evaluated this for GPT-2 (1.5B) and found that this actually makes performance worse, dropping accuracy to 5.3%5.3\%. We hypothesize that the drop in accuracy from using scratch space arises from a snowballing effect, in which partially generated “solutions” with mistakes can derail subsequent generated text. Nevertheless, when generation becomes more reliable and models no longer confuse themselves by their own generations, our dataset’s solutions could in principle teach models to use scratch space and attain higher accuracy.

Examples. We can also qualitatively assess the step-by-step solutions that the model generates. We show examples of generated solutions in Figures 3 and 4. We find that the model can consistently generate correct LaTeX and often performs steps that appear related to the question at hand, but still makes many logical mistakes, both in terms of what the question seems to be asking and in individual steps that are part of a larger derivation.

The Benefits of MATH Solutions. We find that giving models partial step-by-step MATH solutions during inference can improve accuracy. We test performance when we allow models to predict the final answer given a “hint” in the form of a portion of the ground truth step-by-step solution. To do so, for this experiment we prompt models with “⟨P⟩\langle P\rangle <Partial Step-by-Step Solution without Final Answer> Final Answer:” during both fine-tuning and evaluation for different partial fractions of the step-by-step solution. This is the same as the default setting when we let models see 0%0\% of the step-by-step solution. When models see “99%99\%” of the solution, they are given the whole step-by-step solution except for the final answer. We show results with GPT-2 (0.7B) for different fractions of the solution in Figure 5. Observe that the model still only attains approximately 40%40\% when given 99%99\% of the solution, indicating room for improvement.

Finally, we also find that providing models with step-by-step during training can further improve performance. We run an ablation by fine-tuning models on MATH with the same setup as before, except that we only show examples with the final answer and no step-by-step solution. If we fine-tune with only the final answer, the GPT-2 (1.5B) accuracy decreases by 0.6%0.6\% to 6.3%6.3\%.

In this paper, we laid groundwork for future research in machine learning for mathematical problem solving. We introduced the MATH benchmark, which enables the community to measure mathematical problem-solving ability. In addition to having answers, all MATH problems also include answer explanations, which models can learn from to generate their own step-by-step solutions. We also introduce AMPS, a diverse pretraining corpus that can enable future models to learn virtually all of K-12 mathematics. While most other text-based tasks are already nearly solved by enormous Transformers, MATH is notably different. We showed that accuracy is slowly increasing and, if trends continue, the community will need to discover conceptual and algorithmic breakthroughs to attain strong performance on MATH. Given the broad reach and applicability of mathematics, solving the MATH dataset with machine learning would be of profound practical and intellectual significance.

References

  1. Amini et al. (2019) Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. ArXiv, abs/1905.13319, 2019.
  2. Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. ArXiv, abs/2108.07732, 2021.
  3. Bansal et al. (2019) K. Bansal, S. Loos, Markus N. Rabe, Christian Szegedy, and S. Wilcox. Holist: An environment for machine learning of higher-order theorem proving (extended version). ArXiv, abs/1904.03241, 2019.
  4. Brown et al. (2020) T. Brown, B. Mann, Nick Ryder, Melanie Subbiah, J. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, G. Krüger, T. Henighan, R. Child, Aditya Ramesh, D. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, E. Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, J. Clark, Christopher Berner, Sam McCandlish, A. Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005.14165, 2020.
  5. Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, J. Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea. Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, F. Such, D. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, I. Babuschkin, S. Balaji, Shantanu Jain, A. Carr, J. Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, M. Knight, Miles Brundage, Mira Murati, Katie Mayer, P. Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. ArXiv, abs/2107.03374, 2021.
  6. Crouse et al. (2019) Maxwell Crouse, I. Abdelaziz, Cristina Cornelio, Veronika Thost, L. Wu, Kenneth D. Forbus, and Achille Fokoue. Improving graph neural network representations of logical formulae with subgraph pooling. ArXiv, abs/1911.06904, 2019.
  7. Devlin et al. (2019) J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
  8. Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don’t stop pretraining: Adapt language models to domains and tasks. ArXiv, abs/2004.10964, 2020.
  9. He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. ArXiv, abs/2006.03654, 2020.
  10. Hendrycks and Gimpel (2017) Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. ICLR, 2017.
  11. Hendrycks et al. (2021) Dan Hendrycks, C. Burns, Steven Basart, Andy Zou, Mantas Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. ICLR, 2021.
  12. Henighan et al. (2020) T. Henighan, J. Kaplan, Mor Katz, Mark Chen, Christopher Hesse, J. Jackson, Heewoo Jun, T. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, B. Mann, A. Radford, Aditya Ramesh, Nick Ryder, D. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling. ArXiv, abs/2010.14701, 2020.
  13. Hernandez et al. (2021) Danny Hernandez, J. Kaplan, T. Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv, 2021.
  14. Hernández-Orallo (2000) J. Hernández-Orallo. Beyond the turing test. Journal of Logic, Language and Information, 9:447–466, 2000.
  15. Hernández-Orallo (1998) Jose Hernández-Orallo. A formal definition of intelligence based on an intensional variant of algorithmic complexity. EIS, 1998.
  16. Huang et al. (2016) D. Huang, Shuming Shi, Chin-Yew Lin, J. Yin, and W. Ma. How well do computers solve math word problems? large-scale dataset construction and evaluation. In ACL, 2016.
  17. Huang et al. (2019) Daniel Huang, Prafulla Dhariwal, D. Song, and Ilya Sutskever. Gamepad: A learning environment for theorem proving. ArXiv, abs/1806.00608, 2019.
  18. Kaplan et al. (2020) J. Kaplan, Sam McCandlish, T. Henighan, T. Brown, Benjamin Chess, R. Child, Scott Gray, A. Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv, abs/2001.08361, 2020.
  19. Khashabi et al. (2020) Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. Unifiedqa: Crossing format boundaries with a single qa system, 2020.
  20. Lample and Charton (2020) Guillaume Lample and Franccois Charton. Deep learning for symbolic mathematics. ArXiv, abs/1912.01412, 2020.
  21. Legg and Hutter (2007) S. Legg and Marcus Hutter. Universal intelligence: A definition of machine intelligence. Minds and Machines, 17:391–444, 2007.
  22. Lewis et al. (2020) M. Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. ArXiv, abs/1910.13461, 2020.
  23. Li et al. (2020) Wenda Li, L. Yu, Yuhuai Wu, and Lawrence C. Paulson. Modelling high-level mathematical reasoning in mechanised declarative proofs. ArXiv, abs/2006.09265, 2020.
  24. Ling et al. (2017) Wang Ling, Dani Yogatama, Chris Dyer, and P. Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In ACL, 2017.
  25. Liu et al. (2020) J. Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. LogiQA: A challenge dataset for machine reading comprehension with logical reasoning. In IJCAI, 2020.
  26. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019.
  27. McAllester (2020) David A. McAllester. MathZero, the classification problem, and set-theoretic type theory. ArXiv, abs/2005.05512, 2020.
  28. Miao et al. (2020) Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. In ACL, 2020.
  29. Murty et al. (2020) Shikhar Murty, Pang Wei Koh, and Percy Liang. Expbert: Representation engineering with natural language explanations. In ACL, 2020.
  30. Polu and Sutskever (2020) Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving. ArXiv, abs/2009.03393, 2020.
  31. Pólya (1945) George Pólya. How to Solve It. Princeton University Press, 1945.
  32. Rabe et al. (2020) Markus N. Rabe, Dennis Lee, K. Bansal, and Christian Szegedy. Mathematical reasoning via self-supervised skip-tree training. arXiv: Learning, 2020.
  33. Radford et al. (2016) A. Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. CoRR, abs/1511.06434, 2016.
  34. Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, M. Matena, Yanqi Zhou, W. Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
  35. Saxton et al. (2019) D. Saxton, Edward Grefenstette, Felix Hill, and P. Kohli. Analysing mathematical reasoning abilities of neural models. ArXiv, abs/1904.01557, 2019.
  36. Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In NIPS, 2014.
  37. Szegedy (2020) Christian Szegedy. A promising path towards autoformalization and general artificial intelligence. In CICM, 2020.
  38. Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, L. Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv, abs/1706.03762, 2017.
  39. Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurIPS, 2019.
  40. Wigner (1960) Eugene Wigner. The unreasonable effectiveness of mathematics in the natural sciences. Communications in Pure and Applied Mathematics, 1960.
  41. Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, 2020.
  42. Wu et al. (2021) Yuhuai Wu, M. Rabe, Wenda Li, Jimmy Ba, Roger B. Grosse, and Christian Szegedy. Lime: Learning inductive bias for primitives of mathematical reasoning. ArXiv, abs/2101.06223, 2021.

In this appendix, we have more comparisons with previous datasets, a discussion of logic and intelligence tests, further AMPS and MATH details, an analysis of model performance as difficulty level changes, and results with the BART architecture.

We compared to ten datasets in the main paper, and now we will further describe plug-and-chug datasets. Dolphin18K (Huang et al., 2016) is one of the first modern datasets in this space and is based on Yahoo! Answers and includes questions such as “help!!!!!!!(please) i cant figure this out!? what is the sum of 4 2/5 and 17 3/7 ?”. MathQA (Amini et al., 2019) builds on AQuA-RAT (Ling et al., 2017) and claims AQuA-RATs “rationales are noisy, incomplete and sometimes incorrect.” MathQA then cleans AQuA-RAT, though cleaning led the dataset size to be reduced by half of an order of magnitude. Miao et al. (2020) analyze MathQA and observe “the annotated formulas of 27% of the problems do not match their labeled answers,” and they obtain 86%86\% accuracy on a cleaned version of MATH-QA. In contrast AMPS is large and clean as questions are algorithmically generated, and our MATH dataset is carefully curated by the competition mathematics community and contains competition-level problems that are difficult.

Figure 6
Figure 6: Difficult natural language tasks such as LogiQA will soon be solved just by making models larger, assuming trends continue. The Transformers in this figure are UnifiedQA (Khashabi et al., 2020) models of various sizes.

While enormous Transformers perform poorly on MATH, they do well on other logic and intelligence tests.

We analyze Transformers on LogiQA (Liu et al., 2020), a task with logical reasoning questions such as “David knows Mr. Zhang’s friend Jack, and Jack knows David’s friend Ms. Lin. Everyone of them who knows Jack has a master’s degree, and everyone of them who knows Ms. Lin is from Shanghai. Who is from Shanghai and has a master’s degree?” As shown in Figure 6, Transformers are improving on LogiQA, so much so that they will attain human-level performance relatively soon, should trends continue.

We also find that Transformers also do well on the C-Test, a pattern completion test that has a 77%77\% correlation with human IQ (Hernández-Orallo, 2000). An example of a problem from C-Test is the sequence “a, a, z, c, y, e, x, _” which has the answer “g.” We regenerated hundreds of C-Test examples to test GPT-3 (175B) in a 5-shot setting. While GPT-3 had abysmal performance when the sequences were letters, converting letters to numbers helped. After changing ‘a’ to 0, ‘b’ to 1, …\ldots, and ‘z’ to 25, accuracy became approximately 40%40\% on the hardest examples (C-Test questions with complexity “13”). For comparison, on these same examples, average humans attained around 20%20\% accuracy (Hernández-Orallo, 2000).

Rendering Graphics. For the first time, our dataset makes it possible for text-based models to process graphical mathematical figures by expressing figures in asymptote code. For example, Figure 7 shows asymptote code and the figure it produces. In short, it is possible to concisely specify many visual mathematics problems with code, sidestepping the complexity of multi-modal models.

Figure 7
Figure 7: Example of asymptote code and the figure it produces.

AMPS Examples. We show concrete examples from AMPS in Figure 8. AMPS is a mixture of examples from Khan Academy and our 100 Mathematica modules.

Figure 8
Figure 8: A Khan Academy problem and solution, followed by the code for a simple Mathematica module used to generate polynomials GCD problems. These problems are available in AMPS.

Contrasting AMPS and DeepMind Mathematics. AMPS has several hundred exercise types or modules (Khan Academy has 693 modules and Mathematica has 100), while DeepMind mathematics (DM) has only a few dozen. We show all Khan Academy modules in Figures 12, 13, 14 and 15. Most DM exercises increase the diversity of problems by simply having a wide range of coefficients and constants. For example, its derivatives module exclusively covers polynomial derivatives with wide-ranging coefficients, while ours covers mixtures of dozens of major analytic functions. DM opts not to cover concepts and subjects such as logarithms and geometry, unlike AMPS. While DM is formatted in plaintext, AMPS is formatted in LaTeX. Finally, while DM solely has final answers, all 693 Khan Academy modules and 37 of our Mathematica modules have full step-by-step solutions.

We break down MATH accuracy by difficulty levels. In Figure 9, we observe that human difficulty and machine difficulty track each other. In Figure 10, we find that accuracy can vary by level and subject substantially. Finally, in Figure 11(a) and Figure 11(b), we analyze the relation between accuracy and problem and solution length, and find that problems with long questions or ground truth solutions indeed tend to be more difficult than problems with short questions or solutions.

Figure 9
Figure 9: Problems that are more difficult for humans are also more difficult for GPT-2.
Figure 10
Figure 10: Accuracy per subject per difficulty level.
Figure

We use BART (Lewis et al., 2020) to determine whether other existing architectures can improve performance. In the main paper we analyzed the performance of various GPT models, which are unidirectional decoder models. Lewis et al. (2020) introduce BART, which has a bidirectional encoder and unidirectional decoder. While T5 has a similar architecture to BART, its tokenizer removes LaTeX symbols, while BART’s tokenizer does not. Hence we use BART in this paper. After pretraining BART-Large (0.4B) on AMPS and fine-tuning BART on MATH, we find that it obtains 4.9%4.9\% on MATH’s test set, which is slightly worse than our smallest GPT-2 model. Consequently models with a bidirectional encoder and unidirectional decoder do not yield marked changes in MATH accuracy.

Because MATH requires a strong mathematical background to perform well on, and a long amount of time to solve problems, we were restricted to assessing six human participants and could not rely on crowdsourcing sites such as Amazon Mechanical Turk. All participants are university students studying computer science. Four of the participants are authors on the paper. The other two participants are friends or acquaintances of the authors, as this survey was only to give a rough sense of human-level performance. All participants gave consent to use their name, though we opt not to include it. Participants had one hour to complete the questions, all without using a calculator. Participants were offered remuneration, though they all offered to volunteer to work on the problems. The instructions and questions used are as follows.

“This research study is being conducted by the Steinhardt Group at UC Berkeley. For questions about this study, please contact Dan Hendrycks at hendrycks@berkeley.edu. In this study, you will have sixty minutes to complete twenty mathematics problems, all without using a calculator. We would like to remind you that participation in our study is voluntary and that you can withdraw from the study at any time.”

  1. 1.

    A 6-sided die is weighted so that the probability of any number being rolled is proportional to the value of the roll. (So, for example, the probability of a 2 being rolled is twice that of a 1 being rolled.) What is the expected value of a roll of this weighted die? Express your answer as a common fraction.

  2. 2.

    The square of 15 is 225. The square of what other number is 225?

  3. 3.

    Find the sum of all values of xx such that ∣x−1∣=7|x-1|=7.

  4. 4.

    The parabolas defined by the equations y=−x2−x+1y=-x^{2}-x+1 and y=2x2−1y=2x^{2}-1 intersect at points (a,b)(a,b) and (c,d)(c,d), where c≥ac\geq a. What is c−ac-a? Express your answer as a common fraction.

  5. 5.

    If a=8a=8, what is the value of (16a23)13\left(16\sqrt[3]{a^{2}}\right)^{\frac{1}{3}}?

  6. 6.

    Let p(x)p(x) be a cubic polynomial such that p(2)=0,p(−1)=0,p(4)=6p(2)=0,p(-1)=0,p(4)=6, and p(5)=8p(5)=8. Find p(7)p(7).

  7. 7.

    Let SS be the set of complex numbers of the form a+bia+bi, where aa and bb are integers. We say that z∈Sz\in S is a unit if there exists a w∈Sw\in S such that zw=1zw=1. Find the number of units in SS.

  8. 8.

    Find the remainder when 1+2+22+23+⋯+21001+2+2^{2}+2^{3}+\cdots+2^{100} is divided by 77.

  9. 9.

    The length of a rectangle is 3x+103x+10 feet and its width is x+12x+12 feet. If the perimeter of the rectangle is 7676 feet, how many square feet are in the area of the rectangle?

  10. 10.

    A European train compartment has six seats. Four of the seats are broken. Wilhelm needs to fill out a form to indicate that there are broken seats. If he randomly checks off four of the seats in the diagram, what is the probability that he marked the correct seats? Express your answer as a common fraction.

  11. 11.

    We have a triangle △ABC\bigtriangleup ABC where AC=17AC=17, BC=15BC=15, and AB=8AB=8. Let MM be the midpoint of ABAB. What is the length of CMCM?

  12. 12.

    If nn gives a remainder of 33 when divided by 77, then what remainder does 2n+12n+1 give when divided by 77?

  13. 13.

    Our club has 2525 members, and wishes to pick a president, secretary, and treasurer. In how many ways can we choose the officers, if individual members are allowed to hold 22, but not all 33, offices?

  14. 14.

    Find the minimum possible value of

    58−42x+149−1401−x2\sqrt{58-42x}+\sqrt{149-140\sqrt{1-x^{2}}}

    where −1≤x≤1-1\leq x\leq 1?

  15. 15.

    Let a,ba,b, and cc be the roots of x3+7x2−11x−2=0x^{3}+7x^{2}-11x-2=0. Find a+b+ca+b+c.

  16. 16.

    Let H\mathcal{H} be the hyperbola with foci at (±5,0)(\pm 5,0) and vertices at (±3,0)(\pm 3,0), and let C\mathcal{C} be the circle with center (0,0)(0,0) and radius 44. Given that H\mathcal{H} and C\mathcal{C} intersect at four points, what is the area of the quadrilateral formed by the four points?

  17. 17.

    If f(x)=x2−2x+1f(x)=x^{2}-2x+1 and g(x)=2x+1g(x)=\sqrt{2x+1} what is the value of f(g(4))−g(f(3))f(g(4))-g(f(3))?

  18. 18.

    Find the value of rr such that 6r2−19r−72r−7=4r−3\frac{6r^{2}-19r-7}{2r-7}=4r-3.

  19. 19.

    For x>0x>0, the area of the triangle with vertices (0,0),(x,0)(0,0),(x,0) and (x,5)(x,5) is 3030 square units. What is the value of xx?

  20. 20.

    Find the units digit of the following within the indicated number base: 4136−2156413_{6}-215_{6}.

Figure 12
Figure 12: Khan Academy modules in our AMPS pretraining dataset (Part 1).
Figure 13
Figure 13: Khan Academy modules in AMPS (Part 2).
Figure 14
Figure 14: Khan Academy modules in AMPS (Part 3).
Figure 15
Figure 15: Khan Academy modules in AMPS (Part 4).

We create and collect various mathematics problems to create MATH and AMPS.

AMPS consists of problems generated with Mathematica and Khan Academy code. Mathematica serves as a calculator and does not copyright its numerical answer outputs, in much the same way that other calculators do not copyright computations such as 52(mod2)5^{2}\pmod{2}. Khan Academy’s exercise framework follows an MIT License. Since we provide attribution, reuse is not restrictive save for attribution requirements.

MATH problems are created by the Mathematical Association of America (MAA). Although we do not commercialize MATH, we should like to demonstrate that we are far from the boundary for action or infringement. For decades, the MAA has not protected its problem IP even from separate organizations which sell MAA problems, such as AoPS. Courts have ruled that this implies the IP rights are permanently forfeited. We raise this point only to demonstrate the extent to which our reuse for research is within the law, because even commercial reuse of MAA problems is within the law and commonplace. Even so, the MATH dataset is not sold and is likely to have no effect on the value of the original problems. This analysis would be pertinent in the hypothetical situation where Fair Use doctrine did not exist, but MATH and AMPS are covered by Fair Use.

For MATH and AMPS, we abide by Fair Use §107: “the fair use of a copyrighted work, including such use by … scholarship, or research, is not an infringement of copyright”, where fair use is determined by “the purpose and character of the use, including whether such use is of a commercial nature or is for nonprofit educational purposes” and “the effect of the use upon the potential market for or value of the copyrighted work.”

We document the dataset within the paper and note that the dataset and code for reproducing results is available at https://github.com/hendrycks/apps. We do not intend for this dataset to train models that help students cheat on mathematics exams. We intend for others to use this dataset in order to better forecast reasoning capabilities.

We bear all responsibility in case of violation of rights. The MATH data, AMPS data, and our open source code are under an MIT license.