Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations

More formally, suppose an eval consists of $n$ independently drawn questions. We write the score on question $i$ as $s_{i}$, decomposing the score into a mean component $x_{i}$ and a zero-mean random component $\epsilon_{i}$:

We refer to $x_{i}$ as the conditional mean and the variance of the random component $\epsilon_{i}$ as the conditional variance, that is, the mean and variance conditional on the selection of question $i$ in the eval. Denote this latter quantity $\sigma_{i}^{2}=\mathrm{Var}(\epsilon_{i})$.

We also write an unconditional version of these numbers (that is, unconditional on the selection of $i$) as

Let $\mu$ represent the unobserved mean value of the super-population with $\mu=\mathbb{E}[s]=\mathbb{E}[x]$. We wish to conduct inference on (that is, the “true” mean eval score) given only the observed question scores $s_{1},\ldots,s_{n}$. Let $\bar{s}=\frac{1}{n}\sum_{i}s_{i}$ represent the average of the observed scores. It follows from the Law of Large Numbers that $\mu$ may be estimated using $\hat{\mu}=\bar{s}$, and from the Central Limit Theorem (C.L.T.) that the standard error of $\hat{\mu}$ can be estimated as

In the special case where $s_{i}\in\{0,1\}$ (a Bernoulli variable), Equation 1 becomes

We note that it is common practice to compute the standard error of the mean by bootstrapping; see, for instance, the OpenAI evals[16] frameworks. But the Central Limit Theorem is applicable to any evals having scores with finite variance and a large number of questions, and so we regard bootstrapping as unnecessary unless a complicated sampling scheme or estimator is being used. We also note that [7] incorrectly estimates all of its standard errors using $\mathrm{SE}_{\rm Bernoulli}$, even when $s_{i}$ takes fractional values, such as with an F1 score. In these cases, $\mathrm{SE}_{\rm Bernoulli}$ will tend to be conservative (too wide) compared to $\mathrm{SE}_{\rm C.L.T.}$. The Inspect framework[12] correctly computes $\mathrm{SE}_{\rm C.L.T.}$ with its built-in stderr() metric.

We suggest reporting the standard error of the mean alongside (beneath) the mean when reporting eval scores. A common practice in other sciences is to report the standard error in parentheses; we suggest emulating this practice. See Table 2 for an example.

A 95% confidence interval may be computed from such a table as

We next consider eval questions that are drawn in groups, or clusters. For instance, DROP[6], QuAC[5], RACE[13], and SQuAD[17] are reading-comprehension evals having multiple related questions about independently selected passages of text, and multilingual evals such as MGSM[18] consist of the same question translated into many languages. Because the inclusion of questions is non-independent, a key assumption of the Central Limit Theorem (or a bootstrap) is violated, and so a naive application of Equation 1 will yield inconsistent standard errors. Here we show how to use clustered standard errors[1], a technique developed in the social sciences, to account for the dependence and correlation structure present in question clusters.

Let $s_{i,c}$ denote the $i$th question score within cluster $c$, and assume that draws of clusters are independent. Continue to denote the mean observed score as $\bar{s}$. The cluster-adjusted standard error of the mean score can be computed as

The clustered standard error acts as a kind of “sliding scale” between cases where scores within a cluster are perfectly correlated (in which case each cluster acts as a single independent observation) and perfectly uncorrelated (in which case the clustered standard error is equivalent to the unclustered case). The intra-cluster correlations (or lack thereof) are captured by the triple summation (over clusters and cross-terms within clusters); for a derivation of Equation 4, see Appendix A.

A suggested format for reporting cluster-adjusted standard errors is presented in Table 3, and some real-world numbers are reported in Table 4. We note that the cluster adjustment in our real-world example is far from trivial (up to 3X). Failure to adjust standard errors for clustered sampling may lead an unsuspecting analyst to suppose that the measurement of the overall eval score is much more precise than it actually is. We therefore advise that the confidence intervals for reading-comprehension evals reported in [7] are likely anti-conservative (too narrow).

The first strategy for reducing the expected conditional variance is to resample the model a number of times, and to compute the standard error using the question-level mean scores from the resamples. Suppose each question is sampled (answered) $K$ times, and the score $s_{i}$ is the mean of these $K$ answer scores. Since the conditional variances are equal for all $K$ answer scores, the overall conditional variance becomes

This relation should clarify the issue of how many times to resample a given question. Once $\mathbb{E}[\sigma_{i}^{2}]/K\ll\mathrm{Var}(x)$, increasing $K$ further will have little effect on the standard error of $\hat{\mu}$.

We work through an example to show how to compute a value for $K$. Suppose scores are binary (0 or 1) and question difficulty is uniformly distributed, $x\sim U[0,1]$. Then $\epsilon_{i}=1-x_{i}$ with probability $x_{i}$ and $\epsilon_{i}=-x_{i}$ otherwise. A bit of integration reveals that $\mathrm{Var}(x)=1/12$ and $\mathbb{E}[\sigma_{i}^{2}]=1/6$. The required relation reduces to $K\gg 2$, or equivalently $2/K\ll 1$. Writing the variance of this estimator with arbitrary $K$ in terms of the estimator with $K=1$, we have

Going from $K=1$ (no resampling of answers) to $K=2$, the total variance is reduced by 1/3. Increasing to $K=4$, we have a variance reduction of 1/2, and setting $K=6$, we reduce variance by 5/9. The upper limit on variance reduction via resampling in this example is 2/3.

Note that computing a pooled standard error across all $KN$ answers will be inconsistent, as multiple answers to the same question would violate the assumption of independent draws. Refer to Section 2.2 for a discussion of questions drawn in related groups.

The second strategy for reducing the expected conditional variance $\mathbb{E}[\sigma_{i}^{2}]$ is to eliminate the term altogether. For language model evals that do not utilize chain-of-thought reasoning, the conditional variance can be removed by analyzing next-token probabilities, rather than evaluating the model’s sampled (or resampled) output.

Consider for instance a multiple-choice eval, and a prompt that induces a model to produce its answer in its first token. If a correct answer is worth 1 and an incorrect answer is worth 0, and the probability of the correct token is denoted $p_{i}$, then $s_{i}=x_{i}=p_{i}$ and $\epsilon_{i}=0$. This yields $\mathrm{Var}(\hat{\mu})=\mathrm{Var}(p)/n$. Using the uniform-difficulty example from the previous section, next-token probabilities will reduce the variance of the estimator by 2/3 (the upper limit achievable via resampling) compared to grading a single sample from each question.

It may be tempting to reduce the “sampling temperature”[10] of the model in order to reduce (or eliminate) the conditional variance. However, we advise against this practice, unless the purpose is to study the model at the new temperature. Besides altering the model’s behavior, adjusting the sampling temperature may simply shift the conditional variance (which can be mitigated using the two techniques above) into the variance of the conditional means (which cannot), or else reduce conditional variance by injecting bias into the estimator. Two short examples will illustrate these points.

Consider a single-token true/false eval where the conditional score means at $T=1$ are uniformly distributed, $x_{T=1}\sim U[0,1]$. As in Section 3.1, $\mathrm{Var}(x_{T=1})=1/12$. But at $T=0$, $x_{T=0}=1\{x_{T=1}>0.5\}$ and the uniform distribution is “rounded” into a Bernoulli distribution with $p=1/2$. So $\mathrm{Var}(x_{T=0})=1/4$. In this case, reducing the sampling temperature, and thereby eliminating the conditional variance, has inadvertently tripled the minimum variance in the score data from 1/12 to 1/4.

In the above case, $\mathbb{E}[x_{T=1}]=\mathbb{E}[x_{T=0}]$, but this does not always hold. Consider a similar (single-token, true/false) case where $x_{T=1}\sim U[1/3,1]$ and (as a consequence) $x_{T=0}$ is rounded to a Bernoulli distribution with $p=1/4$. Then $\mathbb{E}[x_{T=1}]=2/3<\mathbb{E}[x_{T=0}]=3/4$ and $\mathrm{Var}(x_{T=1})=1/27\ll\mathrm{Var}(x_{T=0})=3/16$; that is, not only has the temperature change shifted the expected score, but the variance of the conditional means has increased approximately five-fold.

We therefore recommend a two-pronged variance-reduction strategy. When next-token probabilities are available, and the language model eval can be conducted using next-token probabilities (i.e. without token generation), compute the expected score for each question, and compute the standard error of expected scores across questions. When next-token probabilities are not available, or the answer requires a chain of thought or other complex interaction, choose a $K$ such that $\mathbb{E}[\sigma_{i}^{2}]/K\ll\mathrm{Var}(x)$ and compute the standard error across question-level mean scores. In neither case should the sampling temperature be adjusted for the sake of reducing variance in the scores.

We introduce model subscripts $A$ and $B$ for the remainder of the article. A naive comparison between eval scores can be made by computing the difference between mean eval scores

and an associated standard error

This two quantities can be used to compute the usual 95% confidence interval and z-score

If two models independently report their eval scores and standard errors, it is possible for an analyst to test their difference for statistical significance – even if the two model reports used non-identical random subsets of eval questions.

The naive comparison above misses an opportunity to reduce the standard error when two models evaluate the same set of questions. Let $s_{A-B,i}=s_{A,i}-s_{B,i}$ represent the difference between scores on question $i$, and let $\bar{s}_{A-B}=\bar{s}_{A}-\bar{s}_{B}$ represent the observed average difference. Then we can estimate the standard error of the estimated difference as

This revised standard error can be plugged into Equations 5 and 6 to compute a confidence interval and z-score.

We can compute the reduction in variance achieved with this paired differences test over the unpaired test. First, write out an expression for the variance in the unpaired case

and the paired case

Combining, and recognizing that the cross-model residuals are uncorrelated, we have

We can thus reduce the variance with paired differences as long as the conditional means of the model scores are correlated; that is to say, if the two models have some amount of agreement on which questions are “easy” and which questions are “hard”.

A short calculation will demonstrate the degree of variance reduction that might be expected. Suppose an eval uses next-token probabilities to form continuous scores with zero conditional variance, and that these scores are uniformly distributed for two models over the $[0,1]$ interval. Suppose that the model scores have a correlation coefficient of 0.5. Then $\mathrm{Var}(s_{A})=\mathrm{Var}(s_{B})=1/12$ and $\mathrm{Cov}(x_{A},x_{B})=0.5\sqrt{\mathrm{Var}(s_{A})\mathrm{Var}(s_{B})}=1/24$. In this case, using paired differences will reduce the variance of the estimator by 1/3 in relative terms (that is, from 1/6 to 1/9 in absolute terms).

Because eval question scores are likely to be positively correlated, even across unrelated models, paired differences represent a “free” reduction in estimator variance when comparing two models. We therefore recommend using the paired version of the standard error estimate wherever practicable. We encourage authors of technical reports to include pairwise differences, pairwise standard errors, and score correlations whenever two or more models are being evaluated. Pairwise standard errors may be computed either directly on the differences, or using the single-sample standard errors, the Pearson product-moment correlation, and the relation

A clustered version of the standard error, appropriate for DROP, QuAC, RACE, SQuAD, MGSM, and other evals where questions are drawn in related groups, is directly computable from the differences as

where $s_{A-B,i,c}$ denotes the score difference on the $i$th question within cluster $c$.

A suggested table format for reporting pairwise results is provided in Table 5. A 95% confidence interval on model differences may be computed from the base estimate and standard error, as in Equation 3.

We now possess the analytic tools needed to rigorously answer the questions posed in the Introduction. Using pairwise analysis on all three evals, and ensuring that standard errors were appropriately clustered on MSGM, the numbers in Table 5 would lead us to conclude that the Galleon indeed outperformed Dreadnought on MATH in a statistically significant way – but that the differences on HumanEval and MGSM are indistinguishable from statistical noise. In other words, while a superficial reading of the eval data might have originally tempted us to conclude that Dreadnought was the overall better-performing model, a closer examination of the data would tend to lead the careful analyst to the opposite conclusion.