Reading + AI
LibrarySign in
Open the original paper on arXiv (new tab)

Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Zhichao Liu, Xiyang Wu, Anhao Li
and 7 more authorsChengsong Huang, Ergan Shang, Yue Yu, Ninghao Liu, Lichao Sun, Haitao Mi, LeoweiLiang

arXiv:2607.08964v3 · This workspace stays on this version. Check arXiv for newer versions

Affiliations and author notes
  • [
  • zli12321@umd.edu, zongxiali@global.tencent.com, zhongzhili@global.tencent.com
Source comparison finished

Original formulas, table values and figures are preserved. Comparison can miss differences; use the original PDF to check details.

Reading view checked: 105 math expressions, 0 rendering failures; 8 of 8 images available.

Compared 60 of 60 source passages against 17 of 17 PDF pages.

The HTML source contains failed formatting commands; these were omitted from the reading view. The original sources are retained. (2)

8 of 8 figure, table and graphic images captured. Open original PDF

Abstract

AI agents have become increasingly capable of autonomously completing short, well-specified tasks. However, existing terminal benchmarks largely focus on relatively simple problems that finish within a few minutes and are typically evaluated only by their final outcome. This setup overlooks intermediate progress and partial solutions, leading to sparse reward signals and an incomplete picture of agent capability.

We introduce Long-Horizon-Terminal-Bench, a challenging terminal benchmark of 46 long-horizon tasks spanning nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, but is further decomposed into fine-grained graded subtasks. This design enables dense intermediate rewards and partial credit, allowing evaluation to capture not only whether an agent reaches the final goal, but also how far it progresses on difficult, open-ended workflows.

Tasks in Long-Horizon-Terminal-Bench typically require hundreds of episodes and tens of minutes to hours of execution, stressing long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving. We evaluate 17 frontier models and find that agents consume on average 9.8M tokens per task, with roughly 239 episodes and 88.9 minutes of execution time per run, making Long-Horizon-Terminal-Bench substantially more demanding than prior terminal-based benchmarks. Even the strongest tested model, Grok 4.5, achieves only 28.3%28.3\% pass@1 at a partial-reward threshold of 0.95 and 19.6%19.6\% at a perfect-reward threshold of 1.0, while the mean pass rate across models is just 6.4%6.4\% and 3.2%3.2\% under the two thresholds, respectively. These results reveal substantial headroom for improvement. We further analyze common failure modes and error patterns, and release Long-Horizon-Terminal-Bench to support future progress on robust long-horizon terminal agents.

1]Tencent HY LLM Frontier 2]University of Maryland, College Park 3]University of Georgia 4]Indiana University 5]National University of Singapore 6]University of Minnesota, Twin Cities 7]Washington University in St. Louis 8]Carnegie Mellon University 9]The Hong Kong Polytechnic University 10]Lehigh University † Core contribution. Project page: https://zli12321.github.io/LHTB/

AI agents built on large language models (LLMs) are rapidly improving at autonomous decision making and tool use Du et al. 2026, Chowa et al. 2026. Recent work has demonstrated impressive performance on short, well-scoped tasks such as fixing a code repository issue, completing a coding ticket, or issuing a handful of shell commands Merrill et al. 2026. However, they still cover only a narrow slice of the long-horizon, domain-specific, and practically important workflows that human experts care about in practice [Zhou et al., 2023, Xie et al., 2024, Merrill et al., 2026].

Nowadays, many real workflows are long-horizon: they require agents to execute hundreds of steps, maintain and update plans over tens of minutes to hours, and manage evolving long-context state Wang et al. 2026. Examples include reproducing results from published research papers Song et al. 2026, installing environments from a github repo, auditing complex multimodal datasets, debugging compiler toolchains, or shepherding a multi-stage ML training pipeline. In these settings, agents must repeatedly use terminal command lines to read and write files, run scripts, inspect partial outputs, revise their plans, and recover from mistakes. The outcome is determined not by a single action, but by sustained, coherent progress over a long sequence of decisions Liu et al. 2026.

Existing benchmarks lack fine-grained verifiers to evaluate agents on long-horizon terminal tasks Jimenez et al. 2024. Dominant terminal and software-engineering benchmarks typically evaluate within a few minutes to at most an hour, and grade agents solely on whether the final state satisfies a test suite Desai et al. 2026, Chu et al. 2026. This leaves two gaps. First, short tasks or narrow horizons understate the difficulty of real workflows that require long-horizon navigation, errored step revision, and multi-stage debugging, not just local patching Merrill et al. 2026, Feng et al. 2026. Second, outcome-only grading yields extremely sparse reward signals: an agent that completes most intermediate steps but fails at the final step can receive the same score as one that fails from the beginning Lin et al. 2025. Both issues make it difficult to understand how well current agents can sustain progress on long workflows, or where they fail.

In this paper we introduce Long-Horizon-Terminal-Bench, a benchmark designed explicitly for long-horizon terminal tasks. Long-Horizon-Terminal-Bench consists of 46 tasks spanning nine categories, including experiment reproduction, interactive games, software engineering, multimodal analysis, and scientific computing. Each task is hosted in a containerized terminal environment and is decomposed into graded subtasks with intermediate checks. Rather than asking agents to solve everything or fail, the grader assigns credit for partial progress toward a final goal, providing dense feedback along long trajectories while preserving the realism of everyday tasks.

Long-Horizon-Terminal-Bench targets domains where the completion of the final big goal depends on the optimal and completion of the sub goals, and agents must execute many steps over long time horizons,. Across 17 frontier agents evaluated, rollouts on Long-Horizon-Terminal-Bench average 239 episodes, 9.8M tokens, and 88.9 minutes of wall-clock time before exit under a 90-minute timeout. These numbers are substantially larger than prior terminal and code-editing benchmarks such as Terminal-Bench 2 tasks and SWE-Bench. Yet even in this relaxed grading, current agents struggle: our strongest reported configuration, Grok 4.5, achieves only 28.3% success rate at a 0.95 reward threshold, and the average pass rate across models is 6.4%. This suggests that long-horizon execution remains a central bottleneck for current systems.

We also conduct qualitative and quantitative analyses of agent failures on Long-Horizon-Terminal-Bench. Our analysis shows that current agents often fail not because every local step is wrong, but because they cannot reliably sustain progress, verify completion, and finish long-horizon tasks within budget. Dense rewards make these differences visible by separating timeout-driven incomplete progress from premature stopping and weak self-verification. We release Long-Horizon-Terminal-Bench and the accompanying evaluation harness to support future work on agents that can plan, verify, and execute reliably over long horizons.

Long-Horizon-Terminal-Bench builds on the Terminal-Bench formulation Merrill et al. 2026, where each task is a containerized terminal environment with a natural-language instruction, all relevant files and tools, and a reference solution that can be executed entirely from the command line. Tasks span diverse domains and are explicitly designed so that progress requires continued work over many steps: reading and modifying code and data, running long scripts, inspecting partial outputs, and iteratively debugging.

Unlike Terminal-Bench, which evaluates tasks in a binary solved/unsolved fashion, Long-Horizon-Terminal-Bench is graded through a subtask-based scheme. Each task is decomposed into a small set of semantically meaningful subtasks with their own checks, and agents are scored by their completion rate across these subtasks. This design provides partial credit and a much denser signal about long-horizon performance, revealing how far agents get through complex workflows rather than only whether they reach the final goal.

A Long-Horizon-Terminal-Bench task follows the Terminal-Bench formulation Merrill et al. 2026: it is specified as a Harbor task with (1) a natural-language instruction, (2) a Docker image that defines the terminal environment, (3) a task configuration file, and (4) an oracle implementation or simulator used for grading. The instruction describes the overall long-horizon goal (e.g., reproduce a figure, repair a robotics SLAM pipeline, audit a multimodal dataset) and is the only specification the agent sees. The Docker image contains all relevant assets, code, data, tools, and helper scripts, so that an agent can complete the task entirely from the terminal. Tasks are interactive: after the container starts, the agent issues shell commands, edits files, runs scripts, and inspects intermediate outputs over hundreds of steps until it either succeeds or times out.

While the task format matches Terminal-Bench 2, Long-Horizon-Terminal-Bench is explicitly designed so that a single subtask typically requires many reasoning steps and many minutes to hours of work and dozens to hundreds of distinct operations. Long-horizon structure comes both from the underlying domains (multi-stage ML pipelines, campaign-style games, scientific audits) and from the way we factor goals into a sequence of intermediate targets that must be discovered and executed over time.

Figure 1
Figure 1: The overall structure of long-horizon terminal task with dense reward grading. Partial rewards show how far models are into completing the task.

Each Long-Horizon-Terminal-Bench task is graded by a deterministic grader that runs inside the container at the end of a rollout. Instead of returning a strict single pass/fail, the grader computes scores for a small set of subtasks {s1,…,sK}\{s_{1},\dots,s_{K}\} that together describe the intended workflow. For every subtask sks_{k} it outputs a normalized score rk∈[0,1]r_{k}\in[0,1]; the task reward is

R  =  ∑k=1Kwk rk∑k=1Kwk,R\;=\;\frac{\sum_{k=1}^{K}w_{k}\,r_{k}}{\sum_{k=1}^{K}w_{k}},

where wkw_{k} are non-negative weights (by default all equal, with higher weight on the final goal when needed). Subtasks are chosen to correspond to meaningful intermediate goals in the workflow, and each is evaluated using objective evidence from the final container state, such as files, outputs, test results, or simulator responses.:

- Binary subtasks. Checks are implemented as strict Boolean conditions over the environment state, such as whether all unit tests exit successfully, whether a service responds on the expected port, or whether required experiment scripts run without error. These subtasks return rk∈{0,1}r_{k}\in\{0,1\} depending on whether the corresponding programmatic check passes.

- Continuous or thresholded subtasks. For quantitative targets (e.g., reproducing a figure, matching a metric, or achieving a given speedup) we use continuous scores. Examples include 1.0 if the reproduced metric is within tolerance of the reference, decreasing linearly to 0 as error grows, or fraction of held-out examples on which the model’s predictions match the oracle. This yields meaningful partial credit when an agent gets close but not perfect.

- Episode-aggregating subtasks. Campaign-style tasks (e.g., games or repeated audits) aggregate across episodes. A typical subtask is the fraction of episodes or levels in which the environment’s internal success flag is triggered, or the mean normalized reward reported by the simulator. This measures whether an agent can perform a long-horizon behavior reliably across many episodes, rather than succeeding only once by chance.

The overall task reward RR is used for all reported metrics. A task is counted as resolved for sucess if R≥τR\geq\tau 1^{1}. We also report mean RR across tasks to capture how far agents progress on problems they cannot fully solve.

Figure 2
Figure 2: Task distribution across the LHTB categories. The 46 tasks span nine long-horizon domains, ranging from software and reverse engineering to scientific computing, earth and climate science, and multimodal analysis.

We construct Long-Horizon-Terminal-Bench by instantiating a common recipe across a diverse set of real-world long-horizon workflows Vidgen et al. 2026, Wu et al. 2026. Each task begins with a realistic professional data-processing or engineering problem, such as materials phase-diagram auditing, robotics SLAM benchmark repair, climate NetCDF extreme-event detection, audio–visual event alignment, scientific-figure data reconstruction, satellite flood change detection, microscopy cell-count quality control, or scanned-document table reconstruction. Some of these professional-workflow tasks are adapted from or inspired by prior long-horizon agent benchmarks such as APEX-Agents Vidgen et al. 2026, while others are newly created for Long-Horizon-Terminal-Bench.

For each problem, we build a complete but deliberately broken terminal-only project. Agents must work entirely through the terminal: inspecting code, running commands, and analyzing artifacts such as images, audio, video, tables, and NetCDF files in order to diagnose and repair the core pipeline. Each task includes reproducible public asset-generation scripts, a weak baseline implementation, an official gold solution, a multi-step solve.sh, and a hidden verifier.

To discourage overfitting to superficial public tests, the public checks only validate command-line behavior, file formats, and a small number of simple examples, and they carry relatively low reward weight. Most of the reward is assigned through hidden stress cases that dynamically generate harder inputs and schema variations, such as nested manifests, gzip-plus-base64 wrappers, renamed fields, missing values, injected noise, rotated or cropped images, anomalous frames, and alternative coordinate or time-dimension conventions. As a result, agents cannot obtain high scores by hard-coding outputs or by only patching the public cases; they must implement robust parsing, core algorithms, and end-to-end artifact generation that generalize beyond the visible examples. The official gold solution is required to achieve a verifier score of 1.01.0 on the full hidden evaluation suite.

We calibrate difficulty by repeatedly running Deepseek-V4-Pro DeepSeek-AI 2026 under 1.5-hour time budgets and adjusting task design until the tasks are challenging but still solvable in principle. We generated 120 candidate tasks with quality filtering, where the final benchmark contains 46 tasks implemented in the Harbor format Merrill et al. 2026, all with containerized environments and a shared agent harness.

The 46 tasks in Long-Horizon-Terminal-Bench span 21 high-level categories, including interactive games, multimodal audits, software engineering, systems and performance, experiment reproduction, earth and environment, scientific computing, robotics, materials science, health and medicine, climate science, and chip design (EDA). Figure 2 shows the distribution. No single category dominates the benchmark: the largest groups are interactive games and multimodal audits, with the remaining tasks spread across a broad range of scientific and engineering workflows.

We evaluate models using the Harbor framework with the Terminus-2 agent harness Harbor Framework Team 2026, which interacts with each task environment through a single long-horizon terminal session. Additionally, for GPT-5.3, we instead use Codex OpenAI 2025b2^{2} as the agent harness. These harness frameworks allow us to evaluate how existing frontier models perform on sustained terminal-based workflows.

We benchmark GPT-5.6-sol OpenAI 2025b, GPT-5.5 OpenAI 2025b, GPT-5.4, GPT-5.3 Codex, DeepSeek V4 Pro DeepSeek-AI 2026, Gemini 3.1 Pro Google DeepMind 2026, GLM 5.1 GLM-5-Team et al. 2026, GLM 5.2, Kimi K2.6 Moonshot AI 2026a, Kimi K2.7 Code Moonshot AI 2026b, MiniMax M3 MiniMax 2026, Qwen3.7 Max Qwen Team 2026b, Qwen3.6 Plus Qwen Team 2026a, Doubao Seed 2.1 Pro ByteDance Seed Team 2026, Hy3 Tencent Hunyuan 2026, Grok 4.20 xAI 2026b, and Grok 4.5 xAI 2026a.

For each model we report pass@1 (fraction of tasks whose normalized reward is at least 0.950.95), mean normalized reward across all tasks, number of episodes to complete the task, length of time, and estimated dollar cost and token usage per task.

Figure 3 summarizes pass rate at R≥0.9R\geq 0.9, R≥0.95R\geq 0.95, and R≥1.0R\geq 1.0, along with mean reward across all 46 tasks. Overall, the evaluated frontier models still struggle on long-horizon tasks, even under a dense-grading evaluation that gives credit for partial progress. Grok 4.5 is the strongest reported model, but still resolves only 28.3%28.3\% of tasks (13/46) at R≥0.95R\geq 0.95. GPT-5.6-sol and GPT-5.5 follow at 15.2%15.2\% (7/46) each, with GPT-5.6-sol posting the slightly higher mean reward. MiniMax M3, Kimi K2.7 Code, and DeepSeek V4 Pro follow at 6.5%6.5\% (3/46) each on the full 46-task run. Below is a mid tier at 4.3%4.3\% that includes Qwen3.7 Max, Doubao Seed 2.1 Pro, Gemini 3.1 Pro, GLM 5.1, and GPT-5.3 Codex, while GLM 5.2, Qwen3.6 Plus, GPT-5.4, and Hy3 each resolve only a single task. Kimi K2.6 and Grok 4.20 solve zero tasks at R≥0.95R\geq 0.95, with Grok 4.20 also recording the lowest mean reward (R=0.10R=0.10) among the reported models. Mean normalized rewards follow a similar ordering: Grok 4.5 also leads this metric at R=0.51R=0.51, indicating that many models make partial progress on hard tasks without fully resolving them. The results suggest that long-horizon execution is not just local reasoning task, but remains a bottleneck even for top frontier models.

Figure 3
Figure 3: LHTB leaderboard. Pass@1 at reward thresholds 0.90.9, 0.950.95, and 1.01.0, together with mean normalized reward, for each model under a shared agent setup, shown in a single model order sorted by pass@1 at R≥0.95R\geq 0.95. Pass rate is the fraction of the 46 tasks whose overall reward RR meets a threshold τ\tau, with R≥1.0R\geq 1.0 corresponding to full completion, while mean RR captures partial progress across all tasks. Grok 4.5 achieves the strongest overall performance, resolving 28.3%28.3\% of tasks at R≥0.95R\geq 0.95. Tightening the threshold to R≥1.0R\geq 1.0 collapses most models to zero pass rate, showing that partial-credit evaluation captures progress that a binary threshold misses.

A key contribution of Long-Horizon-Terminal-Bench is dense, subtask-level grading rather than binary pass/fail, which is essential at this difficulty level. Under a strict threshold (R≥1.0R\geq 1.0), 10 of the 17 models pass zero tasks (Figure 3); even at R≥0.95R\geq 0.95, only 6.4%6.4\% of runs pass while 30.8%30.8\% score below 0.050.05 (Figure 4). The remaining 62.8%62.8\% of runs make real partial progress, but binary grading would count them as complete failures by giving them the same score as a run that makes no progress.

Under binary pass/fail evaluation, the leaderboard still collapses into large tie groups; for example, ten models each solve zero tasks at the strict R≥1.0R\geq 1.0 threshold. Mean reward resolves these ties by quantifying how far a model progresses on tasks it does not fully complete. Among the models tied at one solved task at R≥0.95R\geq 0.95, mean reward spans from 0.270.27 for GPT-5.4 to 0.320.32 for GLM 5.2. Pass rate and mean reward are positively but only moderately correlated (Spearman ρ=0.74\rho=0.74), and they can produce different model rankings because they measure different aspects of long-horizon performance: pass rate captures full task completion, whereas mean reward captures sustained partial progress.

Dense grading also reveals near-misses (0.75≤R<0.950.75\leq R<0.95) runs that are close to completion but fail some final checks. These near-misses occur nearly twice as often as passes (9090 vs. 5050), indicating that many runs get very close to fully solving the task, but fail at the last few steps or checks. For example, Kimi K2.6 solves zero tasks at the R≥0.95R\geq 0.95 threshold, yet has five near-misses and a mean reward of 0.250.25. Its strongest near-miss reaches R=0.94R=0.94 on grammar-fuzz-coverage-hunt, and it also comes close on spot-scheduler-traces (R=0.90R=0.90), poc-exploit-craft (R=0.89R=0.89), and nbody-accel-iterative (R=0.89R=0.89). Under binary grading, it would appear indistinguishable from a model that makes no meaningful progress, even though it repeatedly approaches successful completion. As models improve, such near-misses are likely to provide an important early signal of progress, often revealing more clear capability gains before they are reflected in full pass rates.

Figure 4
Figure 4: Distribution of final rewards over all 17×4617\times 46 model–task runs. Only 5050 runs (6.4%6.4\%, green) pass the R≥0.95R\geq 0.95 threshold, while 241241 runs (30.8%30.8\%, gray) make no meaningful progress (R<0.05R<0.05). The remaining 491491 runs (62.8%62.8\%, blue) achieve partial reward but would all be counted as failures under binary pass/fail evaluation. This partial progress is often substantial rather than trivial: 223223 runs (28.5%28.5\%) reach R≥0.5R\geq 0.5, and the mass in [0.85,0.95)[0.85,0.95) (68 runs) is still larger than the number of full passes. Binary grading collapses a wide range of partially successful runs into a single failure bucket, whereas dense subtask-level rewards preserve the signal needed to distinguish model progress at this difficulty level.

Table 1 reports the cost estimate from per-task token usage and public list prices, which Figure 5 shows the pass rate via the Pareto cost–reward frontier. The average costs span from about $3.6 to $26 per task. Although Grok 4.5 achieves the highest pass rate, it does so at only about $11 per task, well below the most expensive models. GPT-5.6-sol and GPT-5.5 form a second tier of strong but substantially pricier models at roughly $21 per task, and GPT-5.4 is the most expensive overall (about $26 per task) with a much lower pass rate. This gap arises because GPT-5.4 is the weaker model yet requires far more episodes per task (302 vs. 208 for GPT-5.5) at comparable per-M token pricing; it therefore has a higher cost while achieving a lower pass rate.

Hy3 anchors the low-cost end of the Pareto frontier at about $3.6\$3.6 per task, while Doubao Seed 2.1 Pro and MiniMax M3 also lie on the frontier, achieving pass@R≥0.95R\geq 0.95 rates of 4.3%4.3\% and 6.5%6.5\% at roughly $5\$5 and $6\$6 per task. Grok 4.5 extends that frontier upward, reaching 28.3%28.3\% pass@R≥0.95R\geq 0.95 at roughly $11 per task. GPT-5.6-sol matches GPT-5.5’s pass rate at similar cost, but neither approaches Grok 4.5’s cost-efficiency. DeepSeek V4 Pro sits just off the frontier, achieving the same 6.5%6.5\% pass@R≥0.95R\geq 0.95 rate as MiniMax M3 at a similar cost. The remaining models, including Kimi K2.7 Code, GPT-5.3 Codex, GLM5.1, Gemini3.1 Pro, Qwen3.7 Max, GLM5.2, Kimi K2.6, and Grok 4.20, form a mid tier or lower tier with moderate to high costs but low resolution rates, indicating that higher inference spending alone is insufficient to guarantee better long-horizon performance.

Table 1
ModelTokens / task (M)Episodes / taskTime / task (min)Avg cost / task ($)
Grok 4.58.9119774.910.95
GPT-5.6-sol4.3221471.321.14
GPT-5.54.1620872.921.00
MiniMax M320.2031490.06.13
Kimi K2.7 Code8.5418385.48.67
DeepSeek V4 Pro14.4532183.66.19
Qwen3.7 Max6.1321883.57.78
Doubao Seed 2.1 Pro5.8018391.75.16
Gemini 3.1 Pro3.5514885.07.44
GLM 5.15.8412092.65.46
GPT-5.3 Codex4.9930584.68.16
GLM 5.28.4319589.311.93
Qwen3.6 Plus8.6719488.64.47
Hy325.33435176.13.62
GPT-5.410.9030279.325.77
Kimi K2.610.2718892.59.94
Grok 4.2015.9533169.519.64
Average9.7923988.910.79
Table 1: Estimated cost per model on LHTB. On average, models spend 239 episodes and 88.9 minutes per task, at an estimated $10.8 per task. Costs are estimated from per-task token usage and public list prices (USD per million tokens) as of June 2026. Tokens/task counts both input and output tokens; all prompt tokens are billed at the full input rate, since prompt-cache discounts are not guaranteed across providers.
View original tableOriginal Table 1
Figure 5
Figure 5: Cost–reward frontier on LHTB. Estimated total cost (log scale) against pass rate at R≥0.95R\geq 0.95; the dashed line traces the Pareto frontier, and models on the frontier are shown in bold. Grok 4.5 achieves the highest pass rate among the reported models while remaining on the Pareto frontier, and Hy3 stays at the low-cost end of that frontier.

Figure 6
Figure 6: Composition of unresolved runs per model. Each bar decomposes a model’s unresolved tasks (R<0.95R<0.95, out of 46) into timeouts (the agent is still working when the 90-minute budget expires), early exits (the agent terminates on its own with no harness error), and harness errors (non-timeout exceptions such as API- or verifier-side failures). The label inside each timeout segment shows the number of timed-out runs and their mean reward; the label inside each early-exit segment shows the number of early exits and their mean reward. Timeouts dominate for every model (79%79\% of all unresolved runs), showing that many failures reflect incomplete progress rather than immediate breakdowns, while the smaller early-exit segments highlight cases where agents stop too soon despite not having fully solved the task.

Figure 6 shows unresolved runs by termination cause. Across all models, 79%79\% of unresolved runs (518/660518/660) end because the 90-minute budget expires while the agent is still actively working. However, these timed-out runs are not close to completion, with mean reward ranging only from 0.10 to 0.35 across models. By contrast, early exits that account for only 19%19\% of unresolved runs often indicate overconfidence: the agent stops on its own despite not yet satisfying the hidden verifier. Just 3%3\% are harness errors, failures in the agent–environment loop caused by the evaluation harness itself, rather than by the model stopping early or exhausting its time budget.

The main bottleneck is not local execution correctness, but long-horizon completion. On Terminal-Bench 2.0, most trials complete in under 20 minutes, and trajectory-level error analyses find that many failures are execution-related, including disobeying the specification, repeating steps, and missing termination conditions [Merrill et al., 2026]. On Long-Horizon-Terminal-Bench, the more common failure is different: agents can often string together many locally correct actions, but they cannot reliably turn that progress into a finished artifact before the horizon expires. Short-horizon benchmarks primarily measure whether agents can act correctly; Long-Horizon-Terminal-Bench additionally measures whether they can budget a long horizon, and the latter appears to be the binding constraint.

These results suggest that rankings on Long-Horizon-Terminal-Bench reflect not only task-solving ability, but also time efficiency: how much reward a model can accumulate within a fixed budget. As a result, improvements that reduce redundant exploration, preserve state more effectively, or avoid repeated verification loops may yield larger gains than comparable improvements in single-step reasoning alone.

Timeouts dominate model failures, but they are not the full story. Among the 124 runs in which an agent terminates on its own before the budget expires, a different limitation becomes clear: weak self-verification.

We define a false finish as an early exit at relatively high reward, where the agent stops despite having not yet satisfied the hidden verifier. We identify 14 such runs with R≥0.75R\geq 0.75. In these cases, the agent has completed most of the task, judges itself finished, and exits with substantial time remaining. For example, Kimi K2.7 Code stops on duckdb-optimizer-closure at R=0.92R=0.92, GLM 5.2 stops on apex-ib244-matter at R=0.90R=0.90, and seven different models stop on apex-law433-matter between R=0.80R=0.80 and 0.870.87, each with roughly 20 minutes still remaining. These are not hopeless attempts, but near-complete ones in which the agent stops just short of satisfying the hidden verifier.

Dense rewards are essential for exposing this pattern. A binary pass/fail metric would collapse all of these runs into the same failure bucket, whereas dense rewards show how far each agent gets before stopping. This reveals a clear axis of variation in stopping judgment (Figure 6). Kimi K2.7 Code’s early exits average R=0.51R=0.51 and MiniMax M3’s average R=0.42R=0.42, indicating that these models tend to stop later than other models (R=0.22R=0.22 to 0.390.39), after completing a substantial fraction of the task. By contrast, Kimi K2.6 exits at an average of only R=0.11R=0.11, abandoning tasks while most of the work remains. Early exits reveal two distinct abilities: whether a model chooses to stop prematurely, and how accurately it can judge how much work remains before the task is truly complete.

This phenomenon is the long-horizon analogue of the verification failures identified in Terminal-Bench 2.0, such as premature termination, weak verification, and missed final checks [Merrill et al., 2026]. After the obvious public errors are fixed, they must actively search for residual defects that no visible signal reveals. The false-finish pattern suggests that current agents systematically overestimate completion and under-invest in final verification. Closing this gap for long-horizon tasks will require not only stronger local reasoning, but also better self-verification and more calibrated stopping decisions.

Recent agent benchmarks have shifted from static question answering and isolated programming exercises toward interactive, execution-grounded tasks in realistic work environments Zhou et al. 2023, Xie et al. 2024. In software engineering, SWE-Bench Jimenez et al. 2024 evaluates repository-level issue resolution on real GitHub projects, while LiveCodeBench Jain et al. 2024 provides contamination-aware evaluation of code generation and related coding abilities such as execution, self-repair, and test-output prediction. Subsequent benchmarks further broaden the scope of software-agent evaluation: SWT-Bench Mündler et al. 2024 studies test generation and validation for real-world bug fixes, and Terminal-Bench Merrill et al. 2026 places agents in containerized terminals to complete realistic technical tasks such as system configuration, model training, and research-code reproduction. More recent efforts push toward substantially harder and longer workflows, including FrontierSWE Chu et al. 2026, SWE-EVO Le et al. 2025, SWE-Marathon Desai et al. 2026, and Edge-Bench, which target expert-level, multi-stage, or ultra-long-horizon tasks. These benchmarks measure progress in coding and terminal agents, but many still primarily measure end-to-end success or failure, providing limited visibility into how much useful progress an agent made before failing. Long-Horizon-Terminal-Bench uses subtask-based grading exposes partial progress even when the final objective is not fully completed.

Recent work has increasingly characterized long-horizon autonomy in terms of the duration of tasks that agents can complete reliably, rather than aggregate performance on a fixed benchmark Kwa et al. 2025, Wijk et al. 2024. METR’s time-horizon analysis Kwa et al. 2025 makes this perspective explicit by estimating the human-time duration of tasks that frontier agents can complete at a fixed success rate. This framing treats horizon length as a first-class capability variable and is grounded in human-calibrated evaluations such as RE-Bench Wijk et al. 2024 and HCAST Rein et al. 2025, which compare agents against human performance on realistic research-engineering, software-engineering, cybersecurity, and reasoning tasks. At the same time, it also highlights a limitation of outcome-only evaluation in long-horizon settings: a binary success signal provides little information about how much progress was made before failure or where execution began to diverge. Recent studies show that long-horizon degradation can arise from the accumulation of execution errors over extended trajectories, even when short-horizon competence appears strong Sinha et al. 2025. Long-Horizon-Terminal-Bench is designed to complement this line of work by combining extended, many-step autonomy tasks with subtask-level grading, thereby enabling a finer-grained characterization of partial progress and failure throughout long-horizon execution.

Process reward models provide supervision over intermediate reasoning steps rather than only final answers, yielding denser signals than outcome-only supervision Lightman et al. 2023, Khalifa et al. 2025. In parallel, rubric-based approaches decompose evaluation into individually assessable criteria or weighted dimensions of quality, making it possible to score complex outputs beyond a single holistic judgment Rao and Callison-Burch 2026, Pan et al. 2026, Tian and others 2026. Most of this literature uses fine-grained signals primarily as training rewards or judge-based supervision, often relying on learned verifiers or LLM evaluators Uesato et al. 2022, Zheng et al. 2023, Li et al. 2025, Yuan et al. 2024. Long-Horizon-Terminal-Bench applies the same core intuition at evaluation time: each task is decomposed into semantically meaningful subtasks, and each subtask is checked by deterministic, environment-grounded graders.

Benchmark outcomes depend not only on the underlying model, but also on the agent harness that mediates its interaction with the environment Yao et al. 2026, Yang et al. 2024, Li et al. 2026, Lin et al. 2026. Open frameworks such as SWE-agent Yang et al. 2024, OpenHands Wang et al. 2024, and the Terminus family introduced alongside Terminal-Bench Merrill et al. 2026 provide reusable interfaces for autonomous repository- and terminal-based work. Opensource agents such as Codex OpenAI 2025a and OpenClaw OpenClaw Contributors 2026 package nontrivial scaffolding around the base model, including prompting, tool orchestration, context management, and action-selection policies.

We introduced Long-Horizon-Terminal-Bench, a benchmark of 46 containerized terminal tasks spanning nine domains and designed to stress long-horizon execution. Unlike outcome-only benchmarks, Long-Horizon-Terminal-Bench uses deterministic, environment-grounded subtasks to provide dense partial-credit signals, allowing evaluation to capture not only whether an agent finishes a task, but also how far it progresses.

Across 17 frontier models under a shared terminus-2 agent, tasks in Long-Horizon-Terminal-Bench require substantial effort, averaging 239 episodes, 9.8M tokens, $10.8 in API cost, and 88.9 minutes per run. Yet the benchmark remains far from saturated: the strongest model, Grok 4.5, achieves only a 28.3%28.3\% pass rate at R≥0.95R\geq 0.95, while the mean pass rate across models is just 6.4%6.4\%. Dense rewards further reveal that many runs make meaningful but incomplete progress, exposing failure patterns that binary pass/fail would miss.

Our analysis suggests that the main bottleneck is not only local reasoning, but reliable long-horizon completion. Current agents often time out after making partial progress, and even voluntary early exits frequently reflect weak self-verification rather than true completion. These results highlight the need for better planning, stronger memory and progress tracking, and more calibrated stopping decisions for long-horizon tasks. We release Long-Horizon-Terminal-Bench and its evaluation harness to support future work on agents that can plan, verify, and execute reliably over long horizons.

References

  1. ByteDance Seed Team (2026) ByteDance Seed Team Seed2.1 officially released: advancing ai productivity. Note: https://seed.bytedance.com/en/blog/seed2-1-officially-released-advancing-ai-productivityOfficial model release announcement (Doubao Seed 2.1 / Seed 2.1 Pro) Cited by: §3.Source
  2. Chowa et al. (2026) S. S. Chowa, R. Alvi, S. S. Rahman, M. A. Rahman, M. A. K. Raiaan, M. R. Islam, M. Hussain, and S. Azam From language to action: a review of large language models as autonomous agents and tool users. Artificial Intelligence Review. Cited by: §1.
  3. Chu et al. (2026) E. Chu, R. Agarwal, A. Thangamuthu, B. Graham, J. Mattern, F. Jiang, P. Cento, S. Jain, M. Abbasi, M. H. Rezaei, G. Wang, A. Zhang, S. Guo, K. Nguyen, D. Liu, A. Bidgoli, A. Dalmia, A. Dankar, A. Vaddela, C. Chen, K. Kumar, K. Vaish, N. Pour, R. Kondra, S. Badiyani, S. Giri, S. Das, S. Gaikwad, S. Shah, V. Dilawari, and V. Agarwal FrontierSWE. Proximal Blog. Note: https://frontierswe.com/blog Cited by: §1, §4.
  4. DeepSeek-AI (2026) DeepSeek-AI DeepSeek-v4: towards highly efficient million-token context intelligence. External Links: 2606.19348, Link Cited by: §2.3, §3.Source
  5. Desai et al. (2026) R. Desai, J. Hu, J. Cabezas, N. Harsola, P. Shukla, R. B. Chaim, A. E. Assadi, O. M. Kamath, F. Faldu, P. Hebbar, et al. SWE-marathon: can agents autonomously complete ultra-long-horizon software work?. arXiv preprint arXiv:2606.07682. Cited by: §1, §4.
  6. Du et al. (2026) S. Du, J. Zhao, J. Shi, Z. Xie, X. Jiang, Y. Bai, and L. He A survey on the optimization of large language model-based agents. ACM Computing Surveys 58 (9), pp. 1–37. Cited by: §1.
  7. Feng et al. (2026) Y. Feng, J. Sun, Z. Yang, J. Ai, C. Li, Z. Li, F. Zhang, K. He, R. Ma, J. Lin, et al. Longcli-bench: a preliminary benchmark and study for long-horizon agentic programming in command-line interfaces. In Findings of the Association for Computational Linguistics: ACL 2026, pp. 29952–29963. Cited by: §1.
  8. GLM-5-Team et al. (2026) GLM-5-Team, :, A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xie, C. Zhu, C. Yin, C. Wang, G. Pan, H. Zeng, H. Zhang, H. Wang, H. Chen, J. Zhang, J. Jiao, J. Guo, J. Wang, J. Du, J. Wu, K. Wang, L. Li, L. Fan, L. Zhong, M. Liu, M. Zhao, P. Du, Q. Dong, R. Lu, Shuang-Li, S. Cao, S. Liu, T. Jiang, X. Chen, X. Zhang, X. Huang, X. Dong, Y. Xu, Y. Wei, Y. An, Y. Niu, Y. Zhu, Y. Wen, Y. Cen, Y. Bai, Z. Qiao, Z. Wang, Z. Wang, Z. Zhu, Z. Liu, Z. Li, B. Wang, B. Wen, C. Huang, C. Cai, C. Yu, C. Li, C. Hu, C. Zhang, D. Zhang, D. Lin, D. Yang, D. Wang, D. Ai, E. Zhu, F. Yi, F. Chen, G. Wen, H. Sun, H. Zhao, H. Hu, H. Zhang, H. Liu, H. Zhang, H. Peng, H. Tai, H. Zhang, H. Liu, H. Wang, H. Yan, H. Ge, H. Liu, H. Chu, J. Zhao, J. Wang, J. Zhao, J. Ren, J. Wang, J. Zhang, J. Gui, J. Zhao, J. Li, J. An, J. Li, J. Yuan, J. Du, J. Liu, J. Zhi, J. Duan, K. Zhou, K. Wei, K. Wang, K. Luo, L. Zhang, L. Sha, L. Xu, L. Wu, L. Ding, L. Chen, M. Li, N. Lin, P. Ta, Q. Zou, R. Song, R. Yang, S. Tu, S. Yang, S. Wu, S. Zhang, S. Li, S. Li, S. Fan, W. Qin, W. Tian, W. Zhang, W. Yu, W. Liang, X. Kuang, X. Cheng, X. Li, X. Yan, X. Hu, X. Ling, X. Fan, X. Xia, X. Zhang, X. Zhang, X. Pan, X. Zou, X. Zhang, Y. Liu, Y. Wu, Y. Li, Y. Wang, Y. Zhu, Y. Tan, Y. Zhou, Y. Pan, Y. Zhang, Y. Su, Y. Geng, Y. Yan, Y. Tan, Y. Bi, Y. Shen, Y. Yang, Y. Li, Y. Liu, Y. Wang, Y. Li, Y. Wu, Y. Zhang, Y. Duan, Y. Zhang, Z. Liu, Z. Jiang, Z. Yan, Z. Zhang, Z. Wei, Z. Chen, Z. Feng, Z. Yao, Z. Chai, Z. Wang, Z. Zhang, B. Xu, M. Huang, H. Wang, J. Li, Y. Dong, and J. Tang GLM-5: from vibe coding to agentic engineering. External Links: 2602.15763, Link Cited by: §3.Source
  9. Google DeepMind (2026) Google DeepMind Gemini 3.1 pro model card. Note: https://deepmind.google/models/model-cards/gemini-3-1-pro/ Cited by: §3.Source
  10. Harbor Framework Team (2026) Harbor: A framework for evaluating and optimizing agents and models in container environments External Links: Document, Link Cited by: §3.SourceSource
  11. Jain et al. (2024) N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica LiveCodeBench: holistic and contamination free evaluation of large language models for code. External Links: 2403.07974, Link Cited by: §4.Source
  12. Jimenez et al. (2024) C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan Swe-bench: can language models resolve real-world github issues?. In International Conference on Learning Representations, Vol. 2024, pp. 54107–54157. Cited by: §1, §4.
  13. Khalifa et al. (2025) M. Khalifa, R. Agarwal, L. Logeswaran, J. Kim, H. Peng, M. Lee, H. Lee, and L. Wang Process reward models that think. arXiv preprint arXiv:2504.16828. Cited by: §4.
  14. Kwa et al. (2025) T. Kwa, B. West, J. Becker, A. Deng, K. Garcia, M. Hasin, S. Jawhar, M. Kinniment, N. Rush, S. Von Arx, et al. Measuring ai ability to complete long tasks. External Links: 2503.14499, Link Cited by: §4.Source
  15. Le et al. (2025) T. Le, M. V. Thai, D. N. Manh, H. P. Nhat, and N. D. Bui SWE-evo: benchmarking coding agents in long-horizon software evolution scenarios. arXiv preprint arXiv:2512.18470. Cited by: §4.
  16. Li et al. (2026) Z. Li, D. Liu, F. Liu, Y. Zhou, X. Wu, J. Chen, J. Xie, X. Wu, and L. Sun COMFYCLAW: self-evolving skill harnesses for image generation workflows. arXiv preprint arXiv:2607.01709. Cited by: §4.
  17. Li et al. (2025) Z. Li, W. Yu, C. Huang, Z. Liang, R. Liu, F. Liu, J. Che, D. Yu, J. Boyd-Graber, H. Mi, et al. Self-rewarding vision-language model via reasoning decomposition. arXiv preprint arXiv:2508.19652. Cited by: §4.
  18. Lightman et al. (2023) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. External Links: 2305.20050, Link Cited by: §4.Source
  19. Lin et al. (2025) H. Lin, X. Tan, Y. Qin, Z. Xu, Y. Shi, Z. Li, G. Li, S. Cai, S. Cai, C. Fu, et al. Cuarewardbench: a benchmark for evaluating reward models on computer-using agent. arXiv preprint arXiv:2510.18596. Cited by: §1.
  20. Lin et al. (2026) M. Lin, J. Wu, Z. Wang, Z. Shi, Y. Sang, B. He, Z. Liu, T. Wei, Z. Wu, Z. Zhang, et al. Harness updating is not harness benefit: disentangling evolution capabilities in self-evolving llm agents. arXiv preprint arXiv:2605.30621. Cited by: §4.
  21. Liu et al. (2026) Y. Liu, Y. Ma, Y. Miao, Y. Li, Y. Xie, X. Yang, Z. Hu, F. Sung, J. Zhang, and B. Hooi Klong: training llm agent for extremely long-horizon tasks. arXiv preprint arXiv:2602.17547. Cited by: §1.
  22. Merrill et al. (2026) M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y. Shin, T. Walshe, E. K. Buchanan, et al. Terminal-bench: benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868. Cited by: §1, §1, §2.1, §2.3, §2, §3.4, §3.5, §4, §4.
  23. MiniMax (2026) MiniMax MiniMax sparse attention. External Links: 2606.13392, Link Cited by: §3.Source
  24. Moonshot AI (2026a) Moonshot AI Kimi k2.6: from code to creation, from one to many. Note: https://www.kimi.com/ai-models/kimi-k2-6Official Moonshot AI model release page (Kimi K2.6) Cited by: §3.Source
  25. Moonshot AI (2026b) Moonshot AI Kimi k2.7 code: open-source 1t agentic coding model. Note: https://kimik2ai.com/k2.7/Kimi K2.7 Code, released June 12, 2026; model ID kimi-k2.7-code Cited by: §3.Source
  26. Mündler et al. (2024) N. Mündler, M. N. Müller, J. He, and M. Vechev Swt-bench: testing and validating real-world bug-fixes with code agents. Advances in Neural Information Processing Systems 37, pp. 81857–81887. Cited by: §4.
  27. OpenAI (2025a) OpenAI Codex. Note: https://github.com/openai/codexOpenAI coding agent / CLI Cited by: §4.Source
  28. OpenAI (2025b) OpenAI GPT-5 system card. Note: https://openai.com/index/gpt-5-system-card/System card; also arXiv:2601.03267 Cited by: §3, §3.Source
  29. OpenClaw Contributors (2026) OpenClaw Note: Open-source agent platform External Links: Link Cited by: §4.Source
  30. Pan et al. (2026) T. Pan, X. Lin, W. Yang, Q. He, S. Chen, L. Qi, W. Xu, H. Feng, B. Xu, and Y. Xiao Rubriceval: a rubric-level meta-evaluation benchmark for llm judges in instruction following. arXiv preprint arXiv:2603.25133. Cited by: §4.
  31. Qwen Team (2026a) Qwen Team Qwen3.6. Note: https://qwen.ai/blog?id=qwen3.6Official Qwen model release blog (Qwen3.6 Plus) Cited by: §3.Source
  32. Qwen Team (2026b) Qwen Team Qwen3.7. Note: https://qwen.ai/blog?id=qwen3.7Official Qwen model release blog (Qwen3.7 Max) Cited by: §3.Source
  33. Rao and Callison-Burch (2026) D. Rao and C. Callison-Burch Autorubric: unifying rubric-based llm evaluation. arXiv preprint arXiv:2603.00077. Cited by: §4.
  34. Rein et al. (2025) D. Rein, J. Becker, A. Deng, S. Nix, C. Canal, D. O’Connel, P. Arnott, R. Bloom, T. Broadley, K. Garcia, et al. Hcast: human-calibrated autonomy software tasks. arXiv preprint arXiv:2503.17354. Cited by: §4.
  35. Sinha et al. (2025) A. Sinha, A. Arun, S. Goel, S. Staab, and J. Geiping The illusion of diminishing returns: measuring long horizon execution in llms. arXiv preprint arXiv:2509.09677. Cited by: §4.
  36. Song et al. (2026) D. Song, H. Zhang, D. Liu, Y. Liu, Z. Li, Z. Yuan, S. Zhang, and L. Sun Dr. claw: an ai research workspace from idea to paper. GitHub. External Links: Link Cited by: §1.Source
  37. Tencent Hunyuan (2026) Tencent Hunyuan Tencent hunyuan 3. Note: https://hunyuan.tencent.com/Official Tencent Hunyuan model site Cited by: §3.Source
  38. Tian et al. (2026) Z. Tian et al. ARCO: adaptive rubric with co-evolution for multi-step llm-based agents. External Links: 2606.21262, Link Cited by: §4.Source
  39. Uesato et al. (2022) J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275. Cited by: §4.
  40. Vidgen et al. (2026) B. Vidgen, A. Mann, A. Fennelly, J. W. Stanly, L. Rothman, M. Burstein, J. Benchek, D. Ostrofsky, A. Ravichandran, D. Sur, et al. APEX-agents. arXiv preprint arXiv:2601.14242. Cited by: §2.3.
  41. Wang et al. (2024) X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, et al. OpenHands: an open platform for ai software developers as generalist agents. External Links: 2407.16741, Link Cited by: §4.Source
  42. Wang et al. (2026) X. J. Wang, H. Bai, Y. Sun, H. Wang, S. Zhang, W. Hu, M. Schroder, B. Mutlu, D. Song, and R. D. Nowak The long-horizon task mirage? diagnosing where and why agentic systems break. arXiv preprint arXiv:2604.11978. Cited by: §1.
  43. Wijk et al. (2024) H. Wijk, T. Lin, J. Becker, S. Jawhar, N. Parikh, T. Broadley, L. Chan, M. Chen, J. Clymer, J. Dhyani, et al. Re-bench: evaluating frontier ai r&d capabilities of language model agents against human experts. arXiv preprint arXiv:2411.15114. Cited by: §4.
  44. Wu et al. (2026) X. Wu, Z. Li, G. Shi, A. Duffy, T. Marques, M. L. Olson, T. Zhou, and D. Manocha Co-evolving llm decision and skill bank agents for long-horizon tasks. arXiv preprint arXiv:2604.20987. Cited by: §2.3.
  45. xAI (2026a) xAI Grok 4.5. Note: https://x.ai/Official xAI site for Grok 4.5 Cited by: §3.Source
  46. xAI (2026b) xAI Grok. Note: https://x.ai/Official xAI site for the Grok model family Cited by: §3.Source
  47. Xie et al. (2024) T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, et al. OSWorld: benchmarking multimodal agents for open-ended tasks in real computer environments. External Links: 2404.07972, Link Cited by: §1, §4.Source
  48. Yang et al. (2024) J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press SWE-agent: agent-computer interfaces enable automated software engineering. External Links: 2405.15793, Link Cited by: §4.Source
  49. Yao et al. (2026) Y. Yao, X. Tan, C. Liu, Y. Li, Z. Wang, W. Yu, Z. Tan, Y. Tian, G. Zhao, L. Sun, et al. Harness-bench: measuring harness effects across models in realistic agent workflows. arXiv preprint arXiv:2605.27922. Cited by: §4.
  50. Yuan et al. (2024) W. Yuan, R. Y. Pang, K. Cho, X. Li, S. Sukhbaatar, J. Xu, and J. Weston Self-rewarding language models. arXiv preprint arXiv:2401.10020. Cited by: §4.
  51. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems 36, pp. 46595–46623. Cited by: §4.
  52. Zhou et al. (2023) S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig WebArena: a realistic web environment for building autonomous agents. External Links: 2307.13854, Link Cited by: §1, §4.Source

Table 2 lists all 46 benchmark tasks using the nine-category taxonomy used throughout the paper. The difficulty label is derived from the task’s mean reward across model results Figure 3: tasks with mean reward at least 0.50.5 are labeled Easy, and the rest are labeled Hard.

Table 2
Task IDCategoryDescriptionDifficulty
2048Interactive gamesPlay 2048 turn-by-turn via a terminal game server, merging tiles to reach the highest tile, scored by replaying recorded moves in a fresh seeded engine.Hard
alp-paper-reproductionResearch reproduction & MLReproduce a calibrated ALP floating-point compression experiment from the paper’s method and sample columns.Hard
apex-ib244-matterAPEX professional workflowsAdvise on taking EdTech firm KSchool private across 38 reactive stages, rebuilding LBO/DCF/comps models for exact figures.Easy
apex-investment-banking-matterAPEX professional workflowsWork a private-equity take-private of Planet Fitness through 19 reactive stages, rebuilding an LBO model and answering exact figures.Hard
apex-law433-matterAPEX professional workflowsAct as outside counsel across four legal matters and 70 reactive stages over ~600 documents, submitting deterministically-graded answers.Easy
apex-management-consulting-matterAPEX professional workflowsWork two management-consulting client matters across 33 reactive stages, rebuilding business-case and cost-savings models for exact answers.Hard
apex-openroad-ibex-signoffSoftware & reverse engineeringClose RTL-to-GDSII signoff on the ibex CPU with OpenROAD/sky130 across reactive stages until timing, DRC, LVS, area, and power pass.Hard
audio-visual-event-alignmentMultimodal & imaging analysisRepair an audio-video sync audit that detects visual events, audio onsets, dropped frames, and per-clip synchronization offsets.Hard
chess-mateInteractive gamesPlay chess as White against a bundled superhuman Stockfish engine, scored on how long the agent survives before being checkmated.Hard
climate-netcdf-extreme-event-auditEarth, climate & energyRepair a climate audit pipeline computing gridded heatwave, precipitation, drought, anomaly, and index metrics over NetCDF datasets.Hard
commit0-multilib-tddSoftware & reverse engineeringReimplement three whole Python libraries from stubs with no provided tests, writing your own suite, graded on hidden real test suites.Hard
dicom-radiology-auditMultimodal & imaging analysisAudit synthetic DICOM CT series geometry, HU conversion, windowing, masks, and lesion measurements.Hard
document-table-layout-reconstructionMultimodal & imaging analysisRepair a scanned-document pipeline that detects table grids, assigns noisy OCR tokens to cells, and emits CSV/JSON/Markdown tables.Hard
duckdb-optimizer-closureSystems, performance & securityIteratively patch DuckDB’s query optimizer to speed up TPC-H while preserving correctness.Hard
epa-swmm-stormwater-regression-auditEarth, climate & energyAudit EPA SWMM stormwater hydraulic and hydrologic simulations against official solver outputs.Hard
epidemic-inverse-control-auditScientific computing & simulationFit and control a multi-city, age-structured SEIR-H-ICU epidemic model.Hard
foldseek-paper-reproductionResearch reproduction & MLReproduce a calibrated Foldseek-style structural protein search experiment from the Nature Biotechnology paper.Hard
gdal-proj-raster-regressionEarth, climate & energyReproduce GDAL/PROJ raster reprojection, coordinate transform, point sampling, and zonal-statistics regression checks.Hard
generals-bot-arenaSoftware & reverse engineeringDevelop a bot for a Generals.io-style grid strategy game by iteratively playing matches and refining its strategy.Easy
grammar-fuzz-coverage-huntSystems, performance & securityIteratively build a grammar-guided fuzzer to maximize code coverage of target parsers.Easy
great-expectations-auditSoftware & reverse engineeringBuild a Great Expectations-based multi-table data-quality and reconciliation audit pipeline.Hard
langchain-version-migrationSoftware & reverse engineeringMigrate a legacy LangChain RAG/router app across a breaking API change with hard-fail compatibility gates.Hard
materials-phase-diagram-auditScientific computing & simulationRepair a materials phase-diagram audit reconstructing phase boundaries, invariant reactions, lever-rule fractions, and convex-hull stability.Hard
matpower-opf-regressionEarth, climate & energyReproduce MATPOWER/PYPOWER power-flow and optimal-power-flow regression checks with grid-constraint diagnostics.Hard
microscopy-cell-count-qc-auditMultimodal & imaging analysisRepair a microscopy QC audit that segments synthetic cell imagery, flags focus/artifact failures, and reports per-tile channel counts.Hard
modflow6-groundwater-regression-auditEarth, climate & energyAudit USGS MODFLOW 6 groundwater simulations against official solver outputs.Hard
nbody-accel-iterativeScientific computing & simulationIteratively optimize an N-body gravitational simulation from O(N^2) to O(N log N) with correctness verification at multiple scales.Easy
nrel-pysam-hybrid-renewables-auditEarth, climate & energyAudit renewable generation and storage dispatch against NREL PySAM official PVWatts outputs.Hard
opensees-seismic-structural-regression-auditScientific computing & simulationAudit seismic structural-dynamics results against official OpenSeesPy finite-element solver outputs.Hard
poc-exploit-craftSystems, performance & securityGenerate proof-of-concept inputs that trigger specific vulnerability classes using sanitizer feedback.Easy
riscv-core-debugSoftware & reverse engineeringLocalize and fix undisclosed injected bugs in an out-of-order RISC-V SystemVerilog CPU until simulation, compliance, and formal checks pass.Easy
robotics-slam-benchmark-repairResearch reproduction & MLRepair a SLAM benchmark audit that composes SE(2) poses, evaluates loop closures, reconstructs landmark maps, and reports trajectory consistency.Hard
rush_hour_campaignLogic & constraint puzzlesSolve a four-stage 6x6 Rush Hour campaign by hand, submitting legal move routes under length caps.Hard
satellite-flood-change-detection-auditMultimodal & imaging analysisRepair a satellite flood change-detection audit that compares before/after imagery, masks clouds, and reports per-tile flood extent.Hard
scientific-figure-data-reconstructionMultimodal & imaging analysisRepair a pipeline that reconstructs plotted series data from scientific figure images, captions, and partial CSV references.Hard
snake_maze_campaignInteractive gamesPlay a deterministic hard Snake variant, eating seeded apples and surviving growing obstacles as long as possible, scored by replaying the move log.Hard
sokobanLogic & constraint puzzlesSolve a ramp of Sokoban puzzles turn-by-turn through a game server, advancing as far as possible, scored by replaying moves in a clean engine.Hard
spice-ephemeris-regressionScientific computing & simulationReproduce NASA SPICE ephemeris, angle, and ground-station visibility regression checks from pinned kernels.Hard
spot-scheduler-tracesSystems, performance & securityImplement and refine a cloud spot-instance scheduling policy that minimizes cost while meeting hard deadlines.Easy
su2-airfoil-regressionScientific computing & simulationReproduce a hardened SU2 aerodynamic regression matrix from official test-case references and solver history artifacts.Hard
sudoku-recoveryLogic & constraint puzzlesRecover seven progressively corrupted hard variant-Sudoku puzzles turn-by-turn, detecting subtle corrupt givens via global constraint reasoning.Hard
super-marioInteractive gamesPlay Super Mario Bros turn-by-turn through a game server, clearing as many levels as possible, scored by replaying frame inputs in a fresh emulator.Hard
tabular-data-feature-covshiftResearch reproduction & MLRecover a sparse high-dimensional signal from low-dimensional observations under covariate shift, with a leakage-audited honesty gate.Hard
unison-paper-reproductionResearch reproduction & MLReproduce a calibrated UNISON fat-tree network simulation experiment from the paper’s method and settings.Hard
unknown-config-semanticsSoftware & reverse engineeringReverse-engineer an undocumented config format from a noisy spec and repair its normalization engine to pass hidden probes across five stages.Hard
vector-db-iterative-buildSystems, performance & securityBuild an approximate nearest-neighbor vector search service from scratch, iteratively optimizing recall and throughput.Hard
Table 2: Task list for Long-Horizon-Terminal-Bench. Each row shows a task ID, its paper-level category, a one-sentence description, and a coarse difficulty label derived from average task reward in Figure 3 (Easy if mean reward ≥0.5\geq 0.5, otherwise Hard).
View original tableOriginal Table 2