Solving Quantitative Reasoning Problems with Language Models

We present Minerva ​​, a language model that achieves strong performance on many quantitative reasoning tasks. Our model is able to process scientific and mathematical questions formulated in natural language, and generate step-by-step solutions using correct LaTeX notation. Figures 1, 2 and 3 show example responses from Minerva to mathematics and physics questions.

Minerva is based on the PaLM general language models Chowdhery et al., (2022) that are further trained on a high-quality dataset containing scientific and mathematical data. In particular, we start with 8B, 62B, and 540B parameter pretrained models, and continue training them on our technical content dataset. We achieve state-of-the-art performance on MATH Hendrycks et al., (2021), GSM8k Cobbe et al., (2021), and a STEM subset of the MMLU Hendrycks et al., (2020) dataset, all of which include math and science questions formulated in natural language. Notably, our models achieve robust performance in a few-shot setting without being explicitly trained on these evaluation datasets.

The main novelty of this paper is a large training dataset that juxtaposes natural language with the correct use of formal mathematical language, such as equations and diagrams. The data is collected from the arXiv preprint server and from web pages that we carefully process to minimise the loss of mathematical content. In addition to establishing new state-of-the-art results, this work establishes a new baseline for the performance achievable on quantitative reasoning benchmarks by increasing data quality and model size.

To further increase the coverage of quantitative reasoning benchmarks, we build a dataset of over 200 undergraduate-level questions in science and mathematics from MIT’s OpenCourseWare (OCW). This provides a measure of our model’s quantitative reasoning abilities in a chain-of-thought context beyond a pure mathematical setting.

Solving quantitative reasoning problems expressed in natural language has been an active area of study (Koncel-Kedziorski et al.,, 2015; Hosseini et al.,, 2014). Prompting language models using scratchpad Nye et al., (2021) or chain-of-thought Wei et al., (2022) solutions can lead them to output step-by-step solutions to unseen problems. The GSM8k work Cobbe et al., (2021) showed that training verifiers to rerank model outputs can lead to improved performance. The original version of GSM8k included special syntax for algebraic calculations, which were processed by a calculator. In this work we focus on self-contained models without access to external tools.

The standard method for evaluating language models on generative tasks is to greedily sample one solution per problem. Recent works (Chen et al.,, 2021; Li et al.,, 2022; Thoppilan et al.,, 2022; Wang et al.,, 2022) have shown that it is advantageous to sample multiple solutions per problem, and then filter those down to a final answer. We find that majority voting (Wang et al.,, 2022) significantly improves performance over greedy decoding.

The work Drori et al., (2021) includes an evaluation of davinci-002, OpenAI’s latest publicly available language model, on a subset of 90 problems from the MATH dataset. Due to the focus on a subset of questions, as well as changes made to the way questions are formatted, it is difficult to directly compare our results with those of Drori et al., (2021). In Section 3, we compare OpenAI davinci-002 with our models under the same experimental conditions.

Our models were trained on a dataset of 38.5B tokens from webpages filtered for mathematical content and from papers submitted to the arXiv preprint server. In addition, the dataset includes general natural language data, which is the same dataset that was used for pretraining PaLM ​​. Our mathematical webpage dataset was constructed by collecting pages that contain mathematical expressions in MathJax format. The pages underwent a cleaning process that removes most HTML tags but preserves mathematical notation, including LaTeX symbols and formatting. The result is that mathematical formulae like $e^{\pi i}+1=0$ or $E=mc^{2}$ are presented in full to the model during training. This procedure makes it possible for the model to perform well on tasks that require calculation and symbolic manipulation. Table 1 provides a breakdown of the training dataset. See Appendix B for more details.

Our approach is to start with the PaLM pretrained decoder-only transformer language models Chowdhery et al., (2022), and further train (finetune) them on our mathematical dataset using an autoregressive objective. Table 2 contains the main model and training hyperparameters. The largest model, with $540$B parameters, was finetuned on $26$B tokens. While this model is highly undertrained compared to the 8B and 62B models, it still achieves superior performance. Additional details can be found in Appendix C.

We mainly focus on few shot evaluation, though see Appendix E.3 for a discussion of finetuned evaluation. For evaluation, we truncate the inputs from the left to $1024$ tokens and we use the model to generate up to $512$ tokens. When sampling once per problem, we sample greedily. When sampling multiple times per problem we use nucleus sampling (Holtzman et al.,, 2019) with temperature $T=0.6$, $p=0.95$. For generative tasks, the model produces a chain-of-thought answer and demarcates a final answer. We evaluate a solution as correct if the final answer matches the ground truth solution, independent of the quality of the chain-of-thought preceding it. To evaluate correctness, we parse the final answers and compare them using the SymPy library (Meurer et al.,, 2017). This is done in order to correctly identify answers that are mathematically equivalent such as $1/\sqrt{3}$ and $\sqrt{3}/3$. See Appendix D.1 for further details.

The existing datasets on which we focus are:

To evaluate the scientific reasoning capabilities of Minerva ​, we harvested a set of STEM problems at the undergraduate level, most of which involve multi-step reasoning, which we refer to in this paper as OCWCourses ​. Using publicly-available course materials offered by MIT (OpenCourseWare), we collected problems with automatically-verifiable solutions (either numeric or symbolically verifiable via SymPy) from courses including “solid-state chemistry”, “information and entropy”, “differential equations”, and “special relativity.” These problems were processed by contractors to be self-contained and to have a clearly-delineated final answer. Problems asking for a proof or open-ended short answer were not included. In total we curated 272 problems, 191 of which have numeric solutions and 81 have symbolic solutions. In Appendix F, we detail the contributions from each course, and the process of converting these course materials into a format suitable for processing by language models. We also provide the text of all problems. We plan to release these as part of an open-source dataset which will be detailed in an upcoming manuscript.

We find that we can considerably outperform greedy decoding by sampling $k>1$ solutions (with a non-zero temperature) and selecting one using majority voting Wang et al., (2022). This consists of grouping predictions with respect to their final answer and selecting the most common answer. We denote this as maj1@k, following Li et al., (2022). A variation of this algorithm, denoted majn@k, involves selecting the $n$ most common answers. Intuitively, the reason majority voting improves performance is that while there are many ways to answer a question incorrectly, there are typically very few ways to answer correctly.

Contrast majority voting with pass@k ​, where a task is considered solved if any single sample solves it out of $k$ samples. See Section 4.2 for more details on pass@k performance. In Appendix E.1, we report on how performance depends on $k$ for different metrics. We find that while pass@k continues to improve as $k$ is increased, majority voting performance saturates faster: $97\%$ of the large $k$ accuracy is achieved at $k=64$ for MATH and $k=16$ for GSM8k. This is likely because majority voting selects the most common answer in the modeled distribution, and the error of this estimate decreases with increasing $k$. This is in contrast to pass@k where the performance improvement comes from the tail of the distribution, which can keep improving as $k$ is increased.

Log-likelihood is another metric that can be used to rerank samples. We found that majority voting performs significantly better than log-likelihood reranking (see Appendix E.2).

In Appendix H, we study the performance of Minerva 540B on simple arithmetic tasks. The model achieves over $80\%$ accuracy on 10-digit addition and over $20\%$ accuracy on 18-digit addition.

To better understand the types of mistakes our models make, we compare the performance of Minerva 8B and Minerva 62B on 216 problems with high confidence majority decisions of both models. Specifically, we selected examples where the top answer received at least $15\%$ of votes, and that either Minerva 8B was correct and Minerva 62B was incorrect (15 samples), or vice versa (201 samples). The categories and examples for each category are described in Appendix I.2.

As shown in Table 4, the prevailing errors of the 8B model were related to incorrect reasoning or calculations. Many of the calculation errors were relatively benign arithmetic mistakes. Solutions that were too short were relatively rare (in these cases, the model immediately produces an incorrect answer without any intermediate reasoning steps). Finally, in a few cases, the model hallucinates an equation or mathematical fact that is not real.

In the samples where the 62B model was incorrect, the dominating failure modes were again incorrect reasoning and incorrect calculations. In summary, we find that the 62B Minerva model retains most of the skills of the 8B model and improves upon both reasoning and calculation robustness.

In our approach to solving quantitative reasoning problems, we are able to automatically verify whether the final answer to a problem is correct, but we do not have an automatic way to verify the model’s chain of reasoning. This leaves open the possibility of false positives: samples which have the correct final answer, but for which the reasoning is incomplete or incorrect.

We selected 100 random questions from MATH (20 per difficulty level), along with answers sampled at zero temperature from the 62B model. We then manually inspected the answers to determine the false positive rate, which is the ratio between number of false positive examples and number of examples for which the final answer is correct; see Table 5. We found that the overall false positive rate is low, though it does increase with difficulty level.

Our focus on pass@1 and majority voting as the primary evaluation metrics is due in part to the fact that they are less susceptible to false positives than pass@k (Li et al.,, 2022). While the pass@256 accuracy is $84.5\%$ for the 62B model, false positives account for part of it. We inspected the samples that failed in majority voting but passed on pass@k due to a single correct answer, and estimate the false positive rate for pass@256 to be 30% among samples selected in this way. After removing false positives, we estimate that the pass@256 accuracy to be bigger than 68%; see Appendix I.3 for details.

We selected the problems for which our 62B parameter model produced a correct answer, and filtered them to the 100 problems with the highest majority vote score, expecting that problems with a high majority vote score are more likely to have been memorized. For each of these question-answer pairs, we compute the BLEU score across chunks of 500 characters in our Math Web Pages dataset (a histogram of the BLEU scores is shown in Appendix Figure 10). We then manually inspect the 250 documents with the highest BLEU scores. While many of the top matches were from homework help sites with math questions and solutions, none of the questions matched the questions in the subset of MATH under consideration. We have included these 250 segments in Appendix J.1. We note that some problems from MATH can be found on the web. Nevertheless, this analysis concludes that these problems did not make it through our data collection process.

To further investigate memorization, we randomly selected twenty problems which the 62B model answered correctly under majority voting. We manually modified each problem either by introducing minor changes to problem wording (framing) or by changing the numbers which appeared in the problem and modifying the solution accordingly. We then compared the accuracy over sampled solutions before and after the modification. Results are shown in Figure 5. In both cases the accuracy before and after modifications are correlated, with no clear bias in favor of the original formulation. This is suggestive of minimal memorization. The modified problems are listed in Appendix J.2.

We seek to detect memorization of solutions by computing BLEU score between ground truth answers and model generated answers. We use the 62B model and analyze $256$ samples per problem in the MATH dataset. First, we compute overlap statistics for all correct samples. We find that 160 out of 5,000 test questions have a sample with a BLEU score greater than or equal to 80 (see Appendix J.3). We note that they tend to be short solutions. To understand the effect of answer similarity on performance, we remove model samples above a certain BLEU score threshold, and recompute the majority vote accuracy. We find that majority vote performance is robust even down to relatively low similarities (see Figure 5), indicating that performance cannot be attributed to model outputs that are very similar to ground truth answers.

Our approach to quantitative reasoning has several limitations. First, we have no automatic way of verifying the correctness of the model’s answers. This is in contrast to formal approaches, for which automatic verification is intrinsic. Second, our model has no access to external tools such as a calculator or a Python interpreter. It is therefore limited in its ability to perform quantitative reasoning tasks that require complicated numerical calculations. Third, because our model was trained on a large amount of data, we have little direct control over the specific capabilities that the model acquired.

Artificial neural networks capable of solving quantitative reasoning problems in a general setting have the potential of substantial societal impact. Minerva ​, while a step in this direction, is still far from achieving this goal, and its potential societal impact is therefore limited. The model’s performance is still well below human performance, and furthermore, we do not have an automatic way of verifying the correctness of its outputs. If these issues could be solved, we expect the impacts of this model to be broadly positive. A direct application could be an accessible and affordable math tutor which could help improve educational inequalities.

The arXiv dataset contains 2M arXiv papers up to February 2021, in LaTeX format. If multiple LaTeX files were present, they were concatenated. Comments were removed, and anything before the first section header or after an appendix/bibliography header was removed. The title and abstract of each paper were added to the document from the arXiv metadata. In order to retain high quality documents and maximize the information per token, papers were filtered out if they were longer than 75k tokens, had on average more than 0.6 tokens per character, had no \section headers, or ended up being empty after processing. The final arXiv dataset after processing includes 1.2M papers totalling $58$GB of data.

We started with a collection of web pages that included the string "<math" or "MathJax-Element-" in the raw HTML, which we used as our filter for pages that that include mathematical content. We considered pages as of January 2022. We then used several heuristics to process the pages. We found empirically that these are sufficient to extract most of the available mathematical content in either LaTeX format or ASCII-math format. The majority of the documents (about 80% of documents) have one of these two formats:

The remaining documents (about 20%) generally have math in MathML format, which we discarded. After extracting the content in any of the previous two forms, we removed all other content that was inside <math> or <span id=MathJax-Element-*> blocks, because these blocks often encode the MathML version of TeX or AsciiMath content. After filtering, processing, and selecting only English documents, the final dataset size is $60$GB.

Extracting and evaluating the correctness of answers to math questions is non-trivial because answers can often be presented in many different ways, both in terms of formatting (e.g. answers can be underlined, or surrounded by a box) and in terms of mathematical content (a large number can be equivalently represented as 1,000 or 1000, answers about currency potentially have the currency symbol attached to them, etc.). Here we describe how final answers are extracted and normalized. After normalization, answers are compared using SymPy (see below). Failing to normalize answers properly will typically lead to falsely identifying correct answers as incorrect (“false negatives”), and therefore to underestimate the model’s accuracy.

We first extract the final answer from the full model response, which potentially includes chain-of-thought reasoning. In the few-shot prompt, we used the format "Final Answer: The final answer is ANSWER. I hope it is correct." for every final answer. We look for this pattern in the model output and extract ANSWER.

We then apply a normalization function to this answer, shown in Listing 1. In order to develop it we manually inspected ground truth targets, samples from Minerva ​, and samples from OpenAI davinci-002. We were especially careful to avoid changes in the format of the ground truth target that might produce false positives.

After applying this normalization function, we checked whether the formatted target and prediction strings are SymPy-equivalent. SymPy equivalence is determined by parsing the answers via
sympy.parsing.latex.parse_latex and then checking whether substracting the two resulting SymPy objects and applying sympy.simplify gives zero. We set a timeout of $5$s when calling sympy.simplify, and labeled strings as nonequivalent if this timeout was exceeded.

For MATH problems, SymPy equivalence improved overall accuracy by around $1\%$. See Table 6 for the accuracies in MATH with only exact string match vs. SymPy equivalence.

Listing 2 shows the 4-shot prompt used when sampling answers to MATH questions. We picked it by choosing $8$ random examples from MATH and selecting examples which did not include Asymptote plotting commands. We chose four examples so that most problems fit within a context length of $1024$, to enable comparisons with a wide range of models.

We study the dependence of performance on the number of generated samples per question on MATH and GSM8k. Table 7 shows results for maj$1\text{@}k$ and maj$5\text{@}k$, and Figure 6 shows the dependence on $k$ for pass@k and majority voting. We observe that while pass@k continues to improve, majority voting saturates quickly.

Table 8 compares majority voting with reranking based on the log-likelihood that the model assigns to each response. We observe that majority voting is significantly better.

Most of our results involve few-shot prompting Minerva on MATH and other datasets on which the model was not explicitly trained. In this section we discuss finetuning our models on the training split of the MATH dataset, and then evaluating on the test split as before. We finetune both the PaLM and Minerva 8B for $3000$ steps with 2048 tokens per batch with batch size $128$ and dropout of 0.1. Similar to Li et al., (2022), we found that the accuracy for PaLM kept improving despite the test loss increasing. We picked the model with the best test accuracy after $50$ training steps.

We finetuned using a few different prompts: A 0-shot prompt, our custom 4-shot prompt, and a prompt containing 4 random examples. Each model was evaluated using the same prompt as was used during finetuning, except for the random prompt model, with was evaluated using the fixed 4-shot prompt that we used for the non-finetuned models.

The results can be found in Table 9. Standard finetuning does not seem to improve the performance of Minerva ​. On the other hand, it does lead to measurable improvements in PaLM ​, though this performance still lagged behind Minerva ​. These results suggest that the marginal utility of supervised finetuning decreases as one improves the quality and diversity of the unsupervised pretraining or unsupervised finetuning dataset.

From Figure 6, we see how majority voting saturates rather quickly at some $k$, while pass@k keeps improving. Here we analyze the asymptotic behavior of majority voting at large $k$.

Let $c_{i}$ denote the sorted number of counts for answer $i$ when we sample $N$ times and let there be a total for $A_{N}$ answers. In other words, $\sum_{i=1}^{A_{N}}c_{i}=N,c_{i}>c_{i+1}$ . We expect that when sampling $k\ll N$ samples, we can model the sampling distribution as a multinomial distribution with probabilities $p_{i}=\frac{c_{i}}{N}$. This approximation will have the error of attributing $p_{i}=0$ to any answer which doesn’t appear in $N$ draws, so we can’t really resolve probabilities smaller than $1/N$. This issue will not matter for our purposes as long as the maximum probability $p_{1}$ is significantly higher than $1/N$.

If we draw $k$ samples from this multinomial distribution, we expect to not be able to identify the majority answer with $95\%$ confidence as long as

For $k=64$, this bound implies that the resolution for $p_{1}-p_{2}$ is $0.25$, but this is a very rough estimate. However, this exercise quantifies why and how majority voting saturates even if pass@k doesn’t.

Another point is that in order to obtain the majority solution with $95\%$ confidence, we need

for $k=64$, we can probe up to $p_{1}>0.06$.

Table 10 shows the breakdown of problems in our dataset by course. See Table 11 for a breakdown of problems by solution type.

Figure 7 shows the instructions provided to our contractor workforce.

We provide the problems in OCWCourses as a separate file.

As with the MATH dataset, special care must be taken in order to correctly extract answers and evaluate them for correctness. Here we describe how final answers are extracted and normalized. See Listing 4 for the code. During dataset creation, contractors annotated all automatically-verifiable solutions as belonging to one of several types: symbolicexpression, symbolicequation, or numeric. For symbolicexpression and symbolicequation answers, our approach is to convert the answer strings into SymPy quantities, and check equality programmatically. For numeric quantities, we first remove any units from the answer string, then convert the answer string to a float. If either numeric quantity is close to zero, our equality condition is that the absolute value of their difference is less than a threshold (0.01) of their mean; otherwise, we use the numpy.isclose() comparison.

As with MATH, we first extract the final answer from the full model response, which potentially includes chain-of-thought reasoning. In the few-shot prompt, we used the format Final Answer: The final answer is ANSWER. I hope it is correct. for every final answer. We look for this pattern in the model output and extract ANSWER.

This is an example of a question we wrote ourselves:

We used the following to label model mistakes:

We include 2 samples for each of the main failure categories. Notice that some categorization depended on human judgement and in some instances more than one label was applied.

Incorrect calculation.

Incorrect reasoning.

We include below one additional example of a reasoning error committed by the 62B model and fixed by the 540B model. The project webpage features more examples comparing these two models.

Misunderstanding of a problem statement.

This label was applied when a human rater thought that an attempted solution may be a part of a mathematically sound solution to a different problem.

Uses of incorrect facts.

We decided to separate this specific kind of a reasoning error — the model provides a consistent reasoning that relies on an additional fact that does not hold in the given context.