LiveBench: A Challenging, Contamination-Limited LLM Benchmark

LiveBench currently consists of 18 tasks across 6 categories: math, coding, reasoning, language, instruction following, and data analysis. Each task falls into one of two types: (1) tasks which use an information source for their questions, e.g., data analysis questions based on recent Kaggle datasets, or fixing typos in recent arXiv abstracts; and (2) tasks which are more challenging or diverse versions of existing benchmark tasks, e.g., from Big-Bench Hard (Suzgun et al., 2023) or IFEval (Zhou et al., 2023a). The categories and tasks included in LiveBench are:

We evaluate dozens of models, including proprietary models as well as open-source models with sizes ranging from 0.5B to 8x22B. We release all questions, code, and model answers, and we welcome community engagement and collaboration. Our codebase is available at https://github.com/livebench/livebench, and our leaderboard is available at https://livebench.ai.

Evaluating the mathematical abilities of LLMs has been one of the cornerstones of recent research in LLMs, featuring prominently in many releases and reports (Reid et al., 2024; OpenAI, 2023; Bubeck et al., 2023). Our benchmark includes math questions of three types: questions modified from recent high school math competitions, fill-in-the-blank questions from recent olympiad competitions, and questions from our new, harder version of the AMPS dataset (Hendrycks et al., 2021).

Our first two math tasks, Competitions and Olympiad, are based on expert human-designed math problems that offer a wide variety in terms of problem type and solution technique. In Competitions, we include questions from AMC12 2023, SMC 2023, and AIME 2024 modifying the prose and the answer order; in Olympiad, we include questions based on USAMO 2024 and IMO 2024, in which the task is to rearrange masked out equations from the solution into the correct order. These questions test problem solving with algebra, combinatorics, geometry, logic, number theory, probability, and other secondary school math topics (Faires & Wells, 2022).

Finally, we release synthetically generated math questions in the AMPS_Hard task. This task is inspired by the math question generation used to create the MATH and AMPS datasets (Hendrycks et al., 2021). We generate harder questions by drawing random primitives, using a larger and more challenging distribution than AMPS across 10 of the hardest tasks within AMPS.

The coding ability of LLMs is one of the most widely studied and sought-after skills for LLMs (Mnih et al., 2015; Jain et al., 2024; Li et al., 2023a). We include two coding tasks in LiveBench: a modified version of the code generation task from LiveCodeBench (LCB) (Jain et al., 2024), and a novel code completion task combining LCB problems with partial solutions collected from GitHub.

The LCB Generation assesses a model’s ability to parse a competition coding question statement and write a correct answer. We include 78 questions from LiveCodeBench (Jain et al., 2024) which has several tasks to assess the coding capabilities of large language models.

The Completion task specifically focuses on the ability of models to complete a partially correct solution—assessing whether a model can parse the question, identify the function of the existing code, and determine how to complete it. We use LeetCode easy, medium, and hard problems from LiveCodeBench’s (Jain et al., 2024) April 2024 release, combined with matching solutions from https://github.com/kamyu104/LeetCode-Solutions, omitting the last 15-70% of each solution and asking the LLM to complete the solution.

The reasoning ability of large language models is another highly benchmarked and analyzed skill of LLMs (Wei et al., 2022; Suzgun et al., 2023; Yao et al., 2024). In LiveBench, we include three reasoning tasks: our harder version of a task from Big-Bench Hard (Suzgun et al., 2023), Zebra puzzles, and spatial reasoning questions.

Web of Lies v2 is an advancement of the similarly named task included in Big-Bench (bench authors, 2023) and Big-Bench Hard (Suzgun et al., 2023). The task is to evaluate the truth value of a random Boolean function expressed as a natural-language word problem. We create new, significantly harder questions by including additional deductive components and several types of red herrings. Next, we include spatial reasoning questions. This set of 50 handwritten questions tests a model’s ability to make deductions about intersections and orientations of common 2D and 3D shapes.

Finally, we include Zebra Puzzles, a well-known reasoning task (Jeremy, 2009) that tests the ability of the model to follow a set of statements that set up constraints, and then logically deduce the requested information. We build on an existing repository for procedural generation of Zebra puzzles (quint t, 2023). Below, we provide an example question from the Zebra Puzzles task.

LiveBench includes three practical tasks in which the LLM assists in data analysis or data science: column type annotation, table join prediction, and table reformatting. Each question makes use of a recent dataset from Kaggle or Socrata.

The first task is to predict the type of a column of a data table. To create a question for the column type annotation task (CTA), we randomly sample a table and randomly sample a column from that table. We use the actual name of that column as the ground truth and then retrieve some samples from that column. We provide the name of all the columns from that table and ask the LLM to select the true column name from those options.

Data analysts often also require a table to be reformatted from one type to another, e.g., from some flavor of JSON to CSV or from XML to TSV. We emulate that task in TableReformat by providing a table in one format and asking the LLM to reformat it into the target format.

Finally, another common application of LLMs in data analysis is performing table joins (Goldbloom, 2024; Liu et al., 2024b; Sheetrit et al., 2024). In the TableJoin task, the LLM is presented with two tables with partially overlapping sets of columns. The LLM is tasked with creating a valid join mapping from the first to the second table.

An important ability of an LLM is its capability to follow instructions. To this end, we include instruction-following questions in our benchmark, inspired by IFEval (Zhou et al., 2023a), which is an instruction-following evaluation for LLMs containing verifiable instructions such as “write more than $300$ words” or “Finish your response with this exact phrase: {end_phrase}.” While IFEval used a list of 25 verifiable instructions, we use a subset of 16 that excludes instructions that do not reflect real-world use-cases. See Appendix Table 3. Furthermore, in contrast to IFEval, which presents only the task and instructions with a simple prompt like “write a travel blog about Japan”, we provide the models with an article from The Guardian (Guardian Media Group, 1821), asking the models to adhere to multiple randomly-drawn instructions while asking the model to complete one of four tasks related to the article: Paraphrase, Simplify, Story Generation, and Summarize. We score tasks purely by their adherence to the instructions.

Finally, we include multiple language comprehension tasks. These tasks assess the language model’s ability to reason about language itself by, (1) completing word puzzles, (2) fixing misspellings while leaving other stylistic changes in place, and (3) reordering scrambled plots of unknown movies.

First, we include the Connections category. Connections is a word puzzle popularized by the New York Times (although similar ideas have existed previously). In this task, we present questions of varying levels of difficulty with 8, 12, and 16-word varieties. The objective of the game is to sort the words into sets of four words, such that each set has a ‘connection’ between them.

Next, we include the Typos task. The idea behind this task is inspired by the common use case for LLMs in which a user asks the LLM to identify typos and misspellings in some written text but to leave other aspects of the text unchanged. We create the questions for this task from recent ArXiv abstracts, which we ensure originally have no typos, by programmatically injecting common human typos into the text. Below is an example question from the Typos task.

Finally, we include the Plot Unscrambling task, which takes the plot synopses of recently-released movies from IMDb or Wikipedia. We randomly shuffle the synopses sentences and then ask the LLM to simply reorder the sentences into the original plot. We find that this task is very challenging for LLMs, as it measures their abilities to reason through plausible sequences of events.

Maintaining a contamination-limited benchmark requires that we update the set of questions over time. We have so far released two updates, and we plan to continue to release updates to add new questions and remove outdated questions. In each update, we replace $\nicefrac{{1}}{{6}}$ of the questions on average, so that the benchmark is fully refreshed roughly every 6 months. We may speed up the turnover rate of questions in the future, based on interest in LiveBench. Each month, we do not release the new questions until one month later, so that the public leaderboard always has $\nicefrac{{1}}{{6}}$ questions that are private. We choose tasks to update based primarily on two factors: (1) the oldest tasks, and (2) the currently easiest tasks. In this way, the questions in LiveBench will stay new and continue to challenge the most capable LLMs. See additional details, as well as a longer discussion on different forms of contamination, in Section A.6.

Our experiments include 40 LLMs total, with a mix of top proprietary models, large open-source models, and small open-source models. In particular, for proprietary models, we include OpenAI models such as o1-preview, chatgpt-4o, and gpt-4o (Brown et al., 2020; OpenAI, 2023), Anthropic models such as claude-3-5-sonnet-20240620, Google models such as gemini-1.5-pro-002 (Reid et al., 2024), and Mistral models such mistral-large-2407 (Jiang et al., 2023).

For open-source models, we include models such as Llama-3.1-405b-instruct, Llama-3.1-70b-instruct (Dubey et al., 2024), deepseek-v2.5, (Liu et al., 2024a), qwen2.5-72b-instruct (Team, 2024b; Yang et al., 2024), command-r-plus-08-2024 (Cohere, 2024; Cohere For AI, 2024), gemma-2-27b-it (Team, 2024a; Team et al., 2024), mixtral-8x22b-instruct-v0.1 (Jiang et al., 2023), and phi-3.5-moe-instruct (Abdin et al., 2024). See Table 4 for a full list of citations.

For all models and tasks, we perform single-turn evaluation with temperature 0, unless otherwise noted in the model card. All models run with their respective templates from our updated version of FastChat (Zheng et al., 2024). We run all open-source models with bfloat16. When running new models, we take care to set up its hyperparameters and chat template as in the model’s example code, and we also double check the outputs to make sure that the inference, as well as our automated parsing functions, are working correctly and fairly. See more details in Section A.4 and Section A.5. For each question, a model receives a score from 0 to 1. For each model, we compute the score on each task as the average of all questions, we compute the score on each of the six categories as the average of all their tasks, and we compute the final LiveBench score as the average of all six categories. In Appendix B, we give additional documentation including average input/output tokens and cost to run LiveBench for each API model.

We compare all 40 models on LiveBench according to the experimental setup described above; see Table 1 and Table 2. We find that o1-preview-2024-09-12 performs the best overall, 6% better than all other models. o1-preview-2024-09-12 substantially outperforms all other models in the data analysis, language, and math categories. The next-best model is claude-3-5-sonnet-20240620, which far outperforms all other models in the coding category (although o1-mini outperforms claude-3.5 in code generation, claude-3.5 has the edge in code completion). o1-mini-2024-09-12 is third overall and is significantly better than all other models in the reasoning category.

The best-performing open-source models are llama-3.1-405b-instruct and qwen2.5-72b-instruct, which virtually tie with each other and outperform gpt-4-turbo. The best-performing small open-source model is phi-3.5-moe-instruct (see Table 2): with only 6.6B active parameters, it outperforms gpt-3.5 and is on par with mixtral-8x22b.

Now we present analyses involving correlation among different categories and tasks. First, we compute the Pearson correlation coefficient among all pairs of categories and tasks in LiveBench (see Figure 2). We find that unsurprisingly, math, coding, and reasoning all correlate with one another. Interestingly, language correlates fairly well with data analysis, likely due to both categories including tasks that require the LLM to output a large part of the prompt that is modified in a specific way (e.g., by fixing typos or changing the table format). Surprisingly, instruction following correlates relatively weakly with all other categories. Among tasks, we see that math comp correlates the highest with the average LiveBench performance, suggesting that this task is the greatest indicator of overall model performance. This is likely due to these being high-quality, diverse mathematical reasoning questions (which we modified to reduce contamination).

Next, in order to see the strengths and weaknesses of each model, we create a scatterplot of each model’s overall LiveBench performance vs. performance on a single category or task (Figure 3). By plotting a best fit line and computing the residuals for each model, we can compute which models are outliers in specific categories – that is, models that are disproportionately stronger in a particular category relative to the best fit line. We see that the o1 and phi series of models are outliers in terms of reasoning (Figure 3, left), while some of the Llama, gemini, and command-r models are outliers in terms of instruction following. We present additional details in Section A.1, including a table of each model’s relative best and worst tasks (computed as the highest and lowest residuals).

Next, we compare LiveBench to two prominent benchmarks, ChatBot Arena (Chiang et al., 2024) and Arena-Hard (Li et al., 2024). In Figure 4, we show a bar plot comparison among models that are common to both benchmarks, and in Figure 6, we compare the performance of these models to a best-fit line. We also compute the correlation coefficient of model scores among the benchmarks: LiveBench has a $0.91$ and $0.88$ correlation with ChatBot Arena and Arena-Hard, respectively.

Based on the plots and the correlation coefficients, we see that there are generally similar trends to LiveBench, yet some models are noticeably stronger on one benchmark vs. the other. For example, gpt-4-0125-preview and gpt-4-turbo-2024-04-09 perform substantially better on Arena-Hard compared to LiveBench, likely due to the known bias from using gpt-4 itself as the LLM judge (Li et al., 2024). We hypothesize that the strong performance of some models such as the gemini-1.5 models on ChatBot Arena compared to LiveBench may be due to having an output style that is preferred by humans. These observations emphasize the benefit of using ground-truth judging, which is immune to biases based on the style of the output.

As described in Section 2.7, we have completed two monthly updates for LiveBench so far. The rank correlation between the original and first update, and the first and second update, are both $>0.997$, showing that the model rankings have stayed consistent. On the other hand, between the original and the most-recent set of questions, the median and mean average scores (among models included in all iterations of the leaderboard) have both dropped by about 1.2%, showing that the benchmark is becoming harder over time, as newly released models become more capable.

AlpacaEval (Li et al., 2023b; Dubois et al., 2023; Dubois et al., 2024), MT-Bench (Chiang et al., 2024), and Arena-Hard (Li et al., 2024) are benchmarks that employ LLM judges on a fixed set of questions. Using an LLM-as-a-judge is fast, relatively cheap, and has the flexibility of being able to evaluate open-ended questions, instruction-following questions, and chatbots. However, LLM judging also has downsides. First, LLMs have biases towards their own answers (Li et al., 2024). In addition, GPT-4 judges have a noticeable difference in terms of variance and favorability of other models compared to Claude judges. Additionally, LLMs make errors. As one example, question 2 in Arena-Hard asks a model to write a C++ program, yet GPT-4 incorrectly judges gpt-4-0314’s solution as incorrect (Li et al., 2024).

ChatBot Arena (Chiang et al., 2024; Zheng et al., 2024) leverages human prompting and feedback. Users ask questions and receive outputs of two randomly selected models and pick which output they prefer. This preference feedback is aggregated into an Elo score for each model. While human evaluation is great for capturing the preferences of a crowd, using a human-as-a-judge has disadvantages. First, human-judging can be labor-intensive, especially for certain tasks included in LiveBench such as complex math, coding, or long-context reasoning problems. Whenever humans are involved in annotation (of which judging is a sub-case), design choices or factors can cause high error rates (Lease, 2011), and even in well-designed human-annotation setups, high variability from human to human leads to unpredictable outcomes (Rashkin et al., 2023).

LiveCodeBench (Jain et al., 2024) also regularly releases new questions and makes use of ground-truth judging. However, it is limited to only coding tasks. The extensive Omni-MATH benchmark Gao & Liu (2024) encompasses numerous math competitions, although using LLM-as-a-judge grading potentially contributes to a degree of contamination or bias in some of the benchmark’s scores; our completely objective correctness-based scoring avoids this concern. The SEAL Benchmark (Scale AI, 2024), uses private questions with expert human scorers, however, the benchmark currently only contains the following categories: Math, Coding, Instruction Following, and Spanish. In Srivastava et al. (2024), the authors modify the original MATH dataset (Hendrycks et al., 2021) by changing numbers in the problem setup. They find declines in model performance for some LLMs, including frontier ones. However, while such work can evaluate LLMs on data that is not in the pretraining set, the data still ends up being highly similar to the kind of data likely seen in the pretraining set. In addition, the hardness of the benchmark remains the same over time.

Finally, we discuss benchmarks that were the basis for tasks in LiveBench. In IFEval (Zhou et al., 2023b), the authors assess how good LLMs are at following instructions by adding one or more constraints in the instruction as to what the output should be. They limit the set of constraints to those in which it can provably be verified that the generation followed the constraints. In Big-Bench (Srivastava et al., 2022), a large number of tasks are aggregated into a single benchmark with the aim of being as comprehensive as possible. Big-Bench-Hard (Suzgun et al., 2022) investigates a subset of Big-Bench tasks that were particularly challenging for contemporaneous models as well as more complex prompting strategies for solving them.

While we attempted to make LiveBench as diverse as possible, there are still additions from which it would benefit. For example, we hope to add non-English language tasks in the future. Furthermore, while ground truth scoring is beneficial in many ways, it still cannot be used for certain use cases, such as ‘write a travel guide to Hawaii’ in which it is hard to define a ground truth. Finally, while we attempted to make all tasks and categories fair for all models, there are still biases due to certain LLM families favoring certain prompt types. We plan to update the prompts (at the start and end of each question) in the future, as new prompt strategies are developed. Similarly, we plan to continue updating the LiveBench leaderboard as new LLMs are released.

Here, we provide more details from Section 3.2. First, we present the Pearson correlation coefficient and std. error for each category (Table 5) and task (Table 6) compared to the overall average LiveBench score, computed using data from all 40 models. This is a supplement to Figure 2. In Table 7, we compute the relative best and worst task for each model, specifically, the tasks with the highest and lowest residuals of the best fit line vs. overall LiveBench performance. In other words, we compute the task that each model most outperforms and underperforms on, relative to a theoretical model with the same overall performance but has balanced performance across each task.

In this section, we ive the details for the ablation study described in Section 3.3. For hard math and reasoning questions, if an LLM struggles to answer the question, then will it also struggle to determine whether or not a given answer to that question is correct? There are some classes of problems for which the answer is surely ‘no’: any problems that are hard to solve by frontier LLMs, yet easy to check whether an answer is correct or not, such as NP-Hard problems. Another exception is that if an LLM judge is given access to the ground truth, then it will (of course) be able to judge whether or not answers are correct. The tasks in our original experiments (AMC, AIME, and Zebra puzzles) may not fit the class of exceptions. By way of a preliminary study of the above suggestion (that LLM judges cannot judge Zebra puzzles and AMC/AIME questions that they cannot solve), we run an experimental test. We use a judge prompt based on the MT-Bench judge prompt, which is duplicated below.

[Instruction] Please act as an impartial judge and evaluate the quality of the response provided by an AI assistant to the user question displayed below. Your evaluation should consider correctness alone. Identify and correct any mistakes. Be as objective as possible. After providing your explanation, you must rate the response as either 1 (correct) or 0 (incorrect) by strictly following this format: “[[rating]]”, for example: “Rating: [[1]]” [Question] question [The Start of Assistant’s Answer] answer [The End of Assistant’s Answer]

We use gpt-4-turbo-2024-04-09 as the judge. We judge the model outputs of both gpt-4-turbo-2024-04-09 and claude-3-opus-20240229.

See Table 8 and Table 9. We find that the error rate for all tasks is far above a reasonable value, indicating that LLM judges are not appropriate for challenging math and logic tasks. However, we note that there may be other experimental setups which could change the result, such as using a more detailed prompt that is tailored to the task of judging hard math and reasoning problems.

In this section, we describe the categories and tasks of LiveBench and the grading methods in more detail.

LiveBench makes use of automatic regex-based grading methods in order to avoid the biases and other downsides of LLM judging, as detailed in Section 1. On the other hand, automated grading methods have two main pitfalls, which we take care to circumvent. The first is that an instruction-following element is added to the task. For example, a reasoning task with a complex answer instruction format tests instruction-following as well as reasoning, rather than pure reasoning. The second is that care must be made to allow for different answer formats so as not to favor a particular LLM (even after giving exact, unambiguous instructions in the prompt, regarding the format of the answer).

When creating questions, we make sure that the prompt is written in such a way that the answer format is fully specified and unambiguous. We typically give an example of the answer format in the prompt, and specify any potential parts that are unclear.

Additionally, our grading methodology is designed to be fully permissive, accommodating all answer formats within reason. As a rule of thumb, if a human reading the LLM’s answer would be able to comprehend the answer (and assuming it is correct), then the answer should be marked correct. For example, in our math category, models can respond with answers in either latex ‘boxed{}’ or ‘fbox{} ’ environments. For multiple choice questions, we ask for the letter answer, but we accept either the letter or the raw answer. The reason for our less-strict judging is because some models may have been instruction-tuned to a particular format, giving them an advantage or disadvantage. We achieved this by using flexible regex patterns that capture different notation styles or response structures. This approach allows us to evaluate models based on their task-specific skills, rather than their ability to follow specific instructions.

In order to ensure a given task is not overly testing instruction following (in addition to its actual category), and also to ensure scoring accuracy and fairness, we regularly manually inspect a random sample of responses which were labeled incorrect for each model. We look for common answer formats that are incorrect and not picked up by the regex-based parser. This allows us to regularly improve our automatic scoring functions to admit as many answer formats as possible, within reason.

We continue to monitor model outputs and update scoring functions as needed to capture answers appropriately. When adding new models or tasks to the benchmark, we re-evaluate and refine scoring methodologies to ensure fairness and accuracy. This ongoing evaluation and maintenance process enables us to maintain the integrity of our benchmark, providing a comprehensive assessment of models’ strengths and weaknesses. By decoupling instruction following capabilities from task-specific evaluations, we can accurately evaluate and compare models across various tasks and categories.

When integrating new models into LiveBench, we conduct thorough quality control to ensure consistent and accurate evaluation. This process guarantees that our benchmark remains reliable and effective in assessing model capabilities.

To ensure the continued relevance and effectiveness of LiveBench, we have implemented a thorough question update policy. That said, we still maintain a degree of flexibility in updating questions, as well as updating the LiveBench policies as a whole, in order to be able to adapt to new developments and trends in the field (such as the release of models that employ search at inference time, or the popularity of a new type of LLM capability). In this section, we give more details on the update policy laid out in Section 2.7.

We point out two different definitions of contamination: (1) test set contamination, and (2) task contamination – or train test distribution similarity.

The first definition is the one that we use in Section 1: when the test data (the questions and answers from the benchmark) are present in the training data itself. This is the definition that is commonly used in the literature, often called ‘data contamination’, ‘data leakage’, ‘test set contamination’, or even simply ‘contamination’ (Oren et al., 2023; Singh et al., 2024; Kalal et al., 2024; Golchin & Surdeanu, 2023a).

The second definition is related to the question: how well do LLMs generalize today? A low level of generalization is, e.g., training on AMC questions, and generalizing the same questions where the order of the answer choices, and other prose in the questions, are changed, while a high level of generalization is, e.g., training on AMC 12A 2023 and testing on AMC 12A 2024. The latter example, despite LLMs already exhibiting some degree of this type of high generalization, is still ‘fair game’ with respect to pretraining and fine-tuning: many LLMs are trained on competitive math problem tasks and then evaluated on new, unseen test problems from the same distribution.

Despite it being accepted practice, we attempt to guard against excessive uses of (2): we do not publicly release any of the code used to generate LiveBench questions, we always modify the generating code when updating the LiveBench questions (making the questions harder), and we do not publicly release the new questions for one month.

We also acknowledge that LiveBench does not fully satisfy (1): while nearly all questions are from June 2024 or more recent, there are some coding questions from November 2023, and the AMC questions have only undergone a low level of modification from their November 2023 version. Therefore, a limited fraction of LiveBench is likely contaminated on all recent LLMs.

We, the authors, bear all responsibility in case of violation of rights. The license of our repository is the Apache License 2.0.

The benchmark is available on HuggingFace at https://huggingface.co/livebench.

We actively maintain and update the benchmark, already having added multiple updates, and we continue to welcome contributions from the community.

Our Code of Conduct is from the Contributor Covenant, version 2.0. See
https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.

We include a datasheet (Gebru et al., 2021) for LiveBench in https://github.com/LiveBench/LiveBench/blob/main/docs/DATASHEET.md.

Here, we give statistics on the number of questions and average number of output tokens per task, and the total cost of running LiveBench with common API models. For the number of questions for each task, as well as the mean and std. dev number of input and output tokens per question for gpt-4-turbo-2024-04-09, see Table 15. Across the 1000 questions in our current problem set, the mean number of input tokens per question was 1612, and the mean number of output tokens was 395. See Table 16 for the price to run GPT and Claude models on LiveBench, as of October 1, 2024. o1-preview-2024-09-12 is the most expensive at $47.87, while claude-3-haiku is the cheapest, at $0.90.