Toward Generalist Autonomous Research via Hypothesis-Tree Refinement

The coordinator updates $\mathcal{T}\!\textit{ree}$ through a repeated six-step procedure: Observe, Ideate, Select, Dispatch, Backpropagate, and Decide. Each step operates on the tree through a narrow interface for adding nodes, dispatching executors, updating node evidence, propagating insights, pruning subtrees, and merging verified branches. The key point is that the LLM policy chooses how to interpret the research state, while all durable state changes are expressed as controlled mutations of the hypothesis tree.

Observe. At the beginning of each cycle, the coordinator re-grounds itself in the current research state by reading a structured projection of $\mathcal{T}\!\textit{ree}$, including active frontier nodes, recently returned evidence, ancestor insights, and the current best artifact $\mathcal{M}_{\mathrm{best}}$. This step makes the tree the authoritative state after context compression and prevents the coordinator from relying on a lossy conversational history.

Ideate. The coordinator selects a parent node and proposes a small set of child hypotheses beneath it. Each child represents a refinement, alternative, or correction of the parent hypothesis and is initialized as a pending node. Unlike free-form brainstorming, ideation is conditioned on accumulated tree evidence: validated insights provide assumptions to build on, pruned nodes provide negative constraints, and recent executor reports suggest which interventions are feasible or under-tested.

Select. The coordinator chooses pending nodes to execute next. Selection balances the expected utility of a hypothesis with the evidence already accumulated around its ancestors and siblings. A direction may be selected because it has strong prior evidence, because its siblings expose an unresolved ambiguity, or because its failure would clarify an important assumption. Thus selection is not merely score maximization; it is frontier control under partial and delayed feedback.

Dispatch. Selected hypotheses are dispatched to independent executors. Each executor materializes its assigned hypothesis in a fresh worktree, evaluates the modified artifact on $\mathcal{E}_{\mathrm{dev}}$, and returns a compact report containing the dev score, factual result, distilled insight, and branch reference. Parallel execution of sibling hypotheses provides comparative evidence within the same research direction, which is useful for later pruning and abstraction.

Backpropagate. When executor reports return, the coordinator writes their evidence into the corresponding leaf nodes and updates insights along the path to the root. The propagated signal is not only a scalar score. It also includes causal attributions, applicability conditions, and reusable lessons extracted from the experiment. A leaf-level observation such as a data-interface mismatch can therefore become a direction-level constraint, and eventually a global prior that shapes future ideation.

Decide. After the tree absorbs the new evidence, the coordinator decides whether to continue expanding a direction, prune a falsified subtree, stop the run, or attempt to merge a candidate branch. Promotion is guarded by a held-out merge gate: the candidate is evaluated on $\mathcal{E}_{\mathrm{test}}$ in a fresh worktree and is merged into $\mathcal{M}_{\mathrm{best}}$ only if it improves over the current best under the objective $\mathcal{O}$. This gate separates exploratory success on $\mathcal{E}_{\mathrm{dev}}$ from verified artifact-level progress.

An executor implements one local experiment for one assigned hypothesis. Given a node $n$, it receives the hypothesis $h_{n}$, relevant ancestor insights, the current best artifact, and the development evaluator $\mathcal{E}_{\mathrm{dev}}$. It then creates an isolated worktree, applies the minimal intervention needed to realize $h_{n}$, runs the evaluator, inspects failures or inactive code paths, and repairs its own implementation when necessary. This local loop may involve multiple edits and reruns, but it remains bound to the assigned hypothesis.

The executor returns exactly the evidence consumed by the coordinator’s tree interface: a comparable dev score for selection, a factual result for future ideation, a distilled insight for backpropagation, and a branch reference for held-out verification. This contract is important. If an executor were allowed to change the hypothesis when the metric stalls, the returned score would no longer be evidence about the assigned node, and ancestor-level insights would become difficult to interpret. By keeping executors hypothesis-bound, Arbor keeps local engineering flexibility while preserving the semantic meaning of tree updates. Algorithm 1 summarizes the full HTR procedure.

The model-training tasks evaluate whether an agent can improve training algorithms under expensive experimental feedback. In Optimizer Design, we use NanoGPT-Bench (Jordan and contributors, 2025), a benchmark for accelerating NanoGPT training. The initial material is the official tuned Muon optimizer baseline distributed with NanoGPT-Bench, and the objective is to reach the target NanoGPT validation loss in as few optimization steps as possible. The development evaluator uses the standard NanoGPT-Bench task during search, while the test evaluator reruns the selected optimizer with two held-out random seeds and reports the average number of steps. In Architecture Design, we use the autoresearch benchmark (Karpathy, 2026). The agent modifies a given LLM training codebase, with the goal of obtaining a lower final loss under a fixed time budget. The test evaluator again averages two held-out random-seed runs.

The harness-engineering tasks evaluate whether an agent can improve the control logic around another agent. In Terminal-Bench 2.0, the initial material is the standard official terminal-agent codebase for Terminal-Bench 2.0 (Merrill et al., 2026), and the objective is to improve pass rate on terminal-based code and shell tasks. We stratify the 89 tasks by difficulty into 36 development tasks and 53 held-out test tasks, rather than optimizing on the full benchmark. In BrowseComp, the initial material is our standard minimal ReAct-style search harness (Yao et al., 2023b; Wei et al., 2025). The objective is to improve answer accuracy on browsing questions; the development and test sets are 50 and 300 non-overlapping BrowseComp questions, respectively.

The data-synthesis tasks evaluate whether an agent can improve a generation pipeline whose output is judged by downstream model behavior. In Search-Agent Data Synthesis, the initial material is a hand-designed pipeline for generating search-agent questions from seed knowledge. Development uses 50 seed items and test uses 100 disjoint seed items. In Math-Reasoning Data Synthesis, the initial material is a hand-designed pipeline for generating AIME-style reasoning problems; development generates 50 problems with 10 seed and test generates 96 problems with 12 seed. Both tasks are scored by the mean $\mathrm{pass@4}-\mathrm{pass@1}$ gap under a strong GPT-5.5-based ReAct evaluator. This metric rewards problems that are not solved immediately but can be solved with additional attempts.

Our evaluation uses two complementary types of benchmarks. The first is the AO Task Suite in Section 5.1, which consists of real research tasks with task-specific materials, objectives, development evaluators, and held-out test evaluators. The second is MLE-Bench Lite, a long-horizon machine learning engineering benchmark derived from MLE-bench (Chan et al., 2024), which allows comparison against established benchmark systems under the official task setup and reporting protocol.

For the real research tasks, we compare against two strong coding-agent baselines: Codex (OpenAI, 2025) using GPT-5.5 and Claude Code (Anthropic, 2025) using Claude Opus 4.6. Each baseline receives the same initial material, objective, evaluator, and resource budget as Arbor, and is allowed to inspect files, edit code, run experiments, and iterate until the budget is exhausted. For MLE-Bench Lite, we compare against reported benchmark systems, including AIDE (Jiang et al., 2025), ML-Master (Liu et al., 2025b) and ML-Master 2.0 (Zhu et al., 2026), AIRA-dojo (Toledo et al., 2025), InternAgent (Team, 2026), R&D-Agent (Yang et al., 2025), Famou-Agent 2.0 (Li et al., 2025), MARS (Chen et al., 2026b), Leeroo (Nadafian et al., 2026), AIBuildAI (Zhang et al., 2026), LoongFlow (Wan et al., 2025), and AI-Scientist-style systems (Lu et al., 2024; Chen et al., 2026a). The baseline numbers in Table 3 are adopted from the official MLE-Bench leaderboard and the AI-Scientist paper Chen et al. (2026a).

We report native task metrics in the main results, using the direction indicated in Table 1. For cross-task averages and ablations, we also report a normalized held-out improvement over the initial material after orienting all metrics so larger is better. For $\Delta$ rows, percentage-valued metrics use absolute changes; non-percentage metrics such as steps and loss use the relative improvement below:

where $\tilde{S}$ is the native score for higher-is-better metrics and the negated native score for lower-is-better metrics. To measure reliability, we run each stochastic method three times and report Avg@3 with standard deviation unless otherwise specified. For MLE-Bench Lite, we report the official benchmark metrics, including valid-submission rate, above-median rate, any-medal rate, and medal breakdown.

Unless otherwise noted, both the coordinator and executors use Claude Opus 4.6 as the backbone model. All real-research-task runs, including Codex (OpenAI, 2025), Claude Code (Anthropic, 2025), and Arbor, use a 48-hour wall-clock limit. To keep the two single-agent baselines running over this long horizon without manual intervention, we launch Codex and Claude Code through their official /goal mode, which lets each agent autonomously sustain a long-running task and avoid mid-trajectory interruptions; Arbor is launched through its own coordinator loop. The default Arbor budget is 20 coordinator cycles with maximum tree depth 2. Executor parallelism is bounded by the available evaluator resources, and all wall-clock time, token usage, and evaluator calls are counted when comparing against baselines. The same prompt-level task description is used across methods; Arbor receives no task-specific search strategy beyond the adapter that runs the evaluator and parses scores. For MLE-Bench Lite, every task is optimized on a single NVIDIA A100 GPU under the official benchmark resource budget.

Arbor obtains the best held-out result on all six tasks, covering three different types of research artifacts: training algorithms, agent harnesses, and data-generation pipelines. The same controller and hypothesis-tree depth are used across these tasks; only the initial material and evaluator are changed. This suggests that the improvement comes from the search procedure itself rather than from task-specific tuning. The gains are also larger than those of single-trajectory coding agents. On BrowseComp, Arbor improves held-out accuracy from $45.33$ to $67.67$, while Codex and Claude Code reach $50.00$ and $53.33$. On Math-Reasoning Data Synthesis, Arbor improves the held-out pass-gap by $19.79$ points, compared with $5.21$ and $7.29$ points for Codex and Claude Code. The baselines can still make progress, but their gains are smaller and less stable across task types. This supports our main hypothesis: in AO, the bottleneck is not only local code editing, but organizing many trials into a coherent exploration process. The cost results in Section 5.8 further show that Arbor achieves these gains without relying on substantially larger token budgets.

Development feedback is useful for guiding exploration, but it is not a reliable admission criterion. Because the agent repeatedly optimizes against $E_{\mathrm{dev}}$, it can overfit to the development split or exploit evaluator-specific patterns. This is especially visible on Terminal-Bench: Claude Code achieves the highest development score ($75.00$), but its held-out score drops to $71.70$; Arbor has a lower development score ($72.22$), but reaches the best held-out score ($77.36$). This gap motivates the held-out merge gate in Arbor. We use $E_{\mathrm{dev}}$ to guide hypothesis search, but promote a candidate artifact only when it improves $E_{\mathrm{test}}$. This separates exploratory feedback from verified progress. It also makes dev/test disagreement informative: a high-dev, low-test candidate is treated not as a success, but as evidence that the current direction may be exploiting the feedback signal rather than producing a transferable improvement.

Table 5 shows that all variants obtain 100% valid submissions, indicating that the ablation gap is not caused by basic execution failure. The difference instead appears in outcome quality. Full Arbor reaches 81.82% Any Medal, compared with 63.64% for w/o tree and 54.54% for w/o insight feedback. The same pattern appears in stronger categories such as Above Median, Silver, and Gold. This suggests that HTR mainly improves later-stage research refinement: once a runnable solution exists, the tree helps the agent decide which directions to extend, revise, or abandon.

Removing insight feedback while keeping the tree causes a larger drop than removing the tree entirely. This result suggests that hierarchy alone is not sufficient. A tree without propagated lessons can still organize experiments syntactically, but it does not provide the semantic memory needed for later decisions. In contrast, full Arbor uses the tree as a substrate for accumulating evidence: leaf-level results are abstracted into direction-level lessons, which then constrain future ideation and selection.

The full system outperforms both ablations, indicating that the two components address different parts of the search problem. The tree defines where competing hypotheses are stored and compared, while insight feedback determines what reusable information is carried forward. Their combination allows Arbor to convert local experimental outcomes into persistent constraints on future search, rather than treating each experiment as an isolated trial.

Across the six completed cost logs, Arbor uses 20.12M–43.19M tokens, a comparable scale to the single-trajectory baselines. Within this budget, Arbor achieves larger held-out gains on most tasks. This suggests that the improvement is not simply due to spending substantially more tokens, but to how the budget is organized: tokens are used to maintain competing hypotheses, run isolated executions, compare evidence, and update the search tree.

Table 5 also shows that many nodes improve the development score, but only a smaller subset are merged. This gap is expected. A dev-improving node may still be worse than the current best artifact, or may overfit the development evaluator and fail to transfer to the held-out test. The merge gate therefore prevents local development gains from being mistaken for artifact-level progress. In this sense, the tree trace records broad exploration, while the held-out gate admits only verified improvements into the final artifact.

Hypothesis refinement in HTR is a deepening of task understanding: early nodes test broad mechanisms, later nodes identify their limits, and ancestor insights summarize these results into constraints for the next round of proposals. This constraint accumulation is the core process-level benefit over flat trial-and-error.

Earlier experiments persistently reduce the arbitrariness of later search, placing mid-to-late improvements on a higher information baseline. The relevant notion of efficiency is whether the same budget produces a less repetitive and more constrained evidence chain, rather than running long enough to stumble onto a result by chance.

As the tree grows, what has been ruled out, validated, and found to have boundary conditions all become priors constraining the next round of proposals. Arbor’s ideas are therefore not isolated guesses but local advances relative to a known task understanding. Together with the refinement and timing results above, this paints the complete picture of HTR as a process mechanism that makes autonomous research cumulative: not more attempts, but less repetitive and more memory-aware search.

Unlike a code agent whose memory is the linear chat transcript, Arbor externalizes its research state into an explicit hypothesis tree (the idea tree), an in-memory object that is the single authoritative record of a run. Each node holds a hierarchical dotted address (e.g. ROOT, 1, 1.1) that encodes its path from the root, its parent_id and children_ids, a once-written hypothesis, a status field tracing the lifecycle pending $\to$ running $\to$ done $\to$ {merged, pruned}, a development score, a factual result, a distilled insight, and a code_ref branch pointer to the artifact. Depth-1 nodes are broad directions and deeper nodes are concrete refinements, so edges encode hypothesis refinement rather than chronological actions. The coordinator never touches the raw structure; it operates the tree only through a small set of typed tools (TreeAddNode, TreeUpdateNode, TreePrune, TreeSetMeta, TreePropagate) and read projections (TreeView). Storing an experiment is thus a controlled mutation: the executor’s structured report is parsed and written into the node’s score/result/insight/code_ref fields, and the tree is serialized to JSON (and rendered to Markdown) after every mutation. Crucially, when a node finishes Arbor backpropagates its insight: propagate_insights walks from the node’s parent to the root and, at each ancestor, an LLM synthesizes the insights of that ancestor’s children into a concise ($<$200-word) summary. Leaf insights describe concrete implementations, parent insights summarize families of interventions, and the root insight maintains a global understanding of the problem. This layerwise abstraction lets the coordinator reason at the right granularity without rereading raw logs. The concrete schema and persistence format are detailed in Section B.5.

Each pending node is executed by an executor dispatched through RunSubagent or RunSubagentParallel. Rather than editing the shared working tree in place, Arbor creates a fresh git worktree branched from the current trunk HEAD for each candidate, so every hypothesis gets a clean, independently recoverable experimental boundary and several executors can edit overlapping files concurrently without corrupting the trunk or one another. At launch the executor is injected with the assigned hypothesis, the ancestor insights along its path, and the task objective, development evaluator $\mathcal{E}_{\mathrm{dev}}$, held-out evaluator $\mathcal{E}_{\mathrm{test}}$, metric direction, data split, and baseline score stored in the tree metadata via TreeSetMeta. Its permission boundary is deliberately narrow: the executor’s tools (Table 7) act only within its own worktree. It cannot read the trunk, inspect sibling branches, mutate the tree, or run $\mathcal{E}_{\mathrm{test}}$. It may repair implementation bugs and choose reasonable engineering details, but it may not swap the assigned hypothesis for a different research direction, which keeps the returned evidence attributable to the node actually tested. The executor reports back four separated fields: the development score $s_{n}$, a factual result $r_{n}$ (raw observations such as errors, curves, and metric breakdowns), a distilled insight $\iota_{n}$ (the causal lesson), and the branch reference $b_{n}$. The coordinator auto-extracts these, updates the node, and decides admission: a result counts as a useful gain only when it improves $\mathcal{E}_{\mathrm{dev}}$ over the trunk by at least the merge threshold, which triggers the GitMergeBranch held-out gate that re-runs $\mathcal{E}_{\mathrm{test}}$ in a separate detached worktree and promotes the artifact only if it strictly beats the current best. Every other outcome is treated as failure evidence rather than noise: the node’s insight and, if the direction is abandoned, its TreePrune reason are aggregated into the constraint view (TreeView(format="constraints")) that conditions the next Ideate step, so negative results actively narrow the search space. Tool-level details of dispatch, score extraction, template-variable substitution ({cwd}, {node_id}), and artifact filtering are given in Section B.4.

A single auto-research run can span hundreds of turns and many hours of evaluation, far beyond one context window, so Arbor adds explicit mechanisms to keep the search productive instead of stalling on early failures or chasing noisy evaluation swings. First, because all durable state lives in the persisted idea tree rather than the transcript, the run survives crashes, agent restarts, and context compression; post-commit context pruning further elides spent Ideate scratch work and loaded skill bodies once a candidate is committed, bounding context growth. Second, long training or evaluation commands are routed through RunTraining, which blocks until completion or timeout while continuously capturing partial metrics, progress logs, and checkpoints, so even a run that times out at 80% of its epochs returns actionable evidence instead of a silent failure. Third, a convergence detector monitors recent score velocity over a sliding window and counts consecutive non-improving experiments, escalating through warn, paradigm_shift, and stop signals; it also flags parent exhaustion when a parent’s recent children all fail to beat the trunk, prompting the coordinator to summarize the failure pattern and open a fresh depth-1 direction rather than over-exploring a dead branch. A meaningful-improvement threshold prevents a single noisy uptick from resetting this signal, balancing premature stagnation against unbounded exploration.

To remain adaptable across research domains without changing the core loop, Arbor exposes two extension surfaces. Skills are markdown documents with YAML frontmatter, discovered by a registry from both the built-in directory and a project-local .research_agent/skills/ override, and loaded on demand via LoadSkill. The Ideate protocol, for instance, loads idea_drafting, first_principles_probe, and fatal_flaw_scan before proposing candidates, then prunes their bodies from context after each commit, so reasoning guidance is injected just-in-time and does not permanently inflate context. Plugins are YAML domain adapters that specialize the system declaratively rather than through code: each plugin can inject domain guidance at six prompt points (coordinator/executor init, ideate, decide, preamble, and workflow), declare an evaluation contract, mark protected paths and required outputs for the merge guard, override runtime configuration through named profiles, and tune convergence thresholds. For example, the mle_kaggle plugin configures a Kaggle/MLE-bench task with its metric direction, evaluation command, protected data paths, and time-budget profiles entirely in YAML. Together, skills and plugins let Arbor adapt to new artifacts and evaluation regimes while keeping the hypothesis-tree machinery and agent contracts unchanged.

The coordinator’s tool set reflects a strict division of labor: it may read any file in the repository and inspect the tree in any projection, but it may never edit code directly. All implementation work is delegated to executors via RunSubagent or RunSubagentParallel. This design is intentional. If the coordinator could edit code directly, it would be tempted to make small local fixes without creating a new tree node, which would break the invariant that every code change is traceable to a hypothesis. Enforcing the edit boundary at the tool level makes this invariant a system property rather than a behavioral expectation.

Table B.4.1 lists the full tool set. The tree tools (TreeView, TreeAddNode, TreeUpdateNode, TreePrune, TreeSetMeta, TreePropagate) are the primary interface through which the coordinator manages the shared research state. The dispatch tools (RunSubagent, RunSubagentParallel) are the boundary through which the coordinator hands off implementation to executors and receives structured evidence back. The merge tool (GitMergeBranch) is the only path through which a candidate branch can be promoted to the current best, enforcing the held-out gate at the tool level rather than relying on the coordinator to remember to verify before merging. Table 6 reports the key hyperparameters used across all experiments.

The executor’s tool set is designed around a single constraint: every action must be local to its worktree. The executor has no tools for reading the current best, inspecting sibling branches, or modifying the shared tree. This isolation is not merely a safety measure. It is what allows multiple executors to run concurrently on overlapping codebases without coordination overhead. Each executor operates as if it has an exclusive copy of the repository, because its worktree is literally a separate checked-out directory that shares the git object store but maintains its own HEAD and index. The RunTraining tool deserves special mention: for experiments that involve multi-hour training runs, using Bash with a timeout would either cut the run short or block the executor for an unbounded duration. RunTraining instead blocks until the command terminates or the configured timeout is reached, continuously capturing intermediate metrics, progress logs, and checkpoints so that even a timed-out run returns actionable partial evidence. This matters in practice: a training run that completes 80% of its epochs before timing out often reveals enough about a direction’s trajectory to inform a principled prune or continuation decision, which is more useful than a silent timeout that leaves the coordinator without evidence.

A recurring challenge in multi-agent systems is ensuring that evaluation commands measure the right code. In Arbor, each executor runs in a different directory, on a different branch, with potentially different data locations. A naively hardcoded evaluation command would silently evaluate the wrong artifact. The template variable system addresses this by requiring the coordinator to specify evaluation commands using two placeholders: {cwd} is substituted with the executor’s worktree path before the command reaches the executor’s prompt, and {node_id} is substituted with the tree node identifier so that results from concurrent experiments are written to distinct output locations and never overwrite one another. The coordinator is also explicitly prohibited from hardcoding absolute paths in evaluation commands. Only {cwd}-relative paths are permitted.

Score extraction from evaluator output follows a two-stage pipeline that handles the diversity of real evaluation scripts. Many harnesses already emit a structured JSON block with a score key. The system attempts to parse this first. If no valid JSON block is found, as is common with training scripts that report loss or accuracy in free-form log lines, a secondary LLM call reads the full command output and extracts the primary metric as a percentage. The distinction between a primary metric and ancillary logging is task-specific, so the LLM call uses a concise system prompt that instructs it to return only a single JSON object and to pick the most prominent performance figure if multiple metrics appear.

The held-out evaluator $\mathcal{E}_{\mathrm{test}}$ is architecturally separated from the development loop. Executors are instructed never to run $\mathcal{E}_{\mathrm{test}}$. It is accessible only through the GitMergeBranch tool, which creates a fresh detached worktree at the candidate branch’s HEAD, applies the same template substitution, and runs eval_cmd_test in complete isolation from the main repository. Configurable retry logic with exponential backoff handles transient infrastructure failures such as gpu allocation timeouts or flaky network reads. The merge is admitted only if the extracted test score strictly exceeds the current best score under the metric direction specified in the task. This two-worktree design uses one worktree for development evaluation inside the executor and one for held-out evaluation inside the merge gate. It ensures that neither the executor nor the coordinator ever evaluates the candidate artifact in the same directory as the current best, eliminating any risk of result leakage between branches.

Before a worktree branch is committed, Arbor applies artifact filtering to separate implementation changes from generated outputs. The system inspects the full worktree diff and classifies changed paths as either implementation files or artifacts. Raw logs, model checkpoints, cache directories, generated data, and large binary files are excluded from the commit, keeping experiment branches compact and ensuring that subsequent three-way merges into the current best branch involve only meaningful code differences. This filtering is conservative by design: a borderline file is included rather than excluded, because a missing implementation file breaks the merge, whereas a spurious large file only adds storage overhead.

The root node anchors the initial artifact and task contract. Depth-1 nodes represent broad research directions, while deeper nodes represent concrete refinements, alternatives, or corrections under their parent. Edges therefore encode hypothesis refinement rather than chronological agent actions. During a run, pending leaves form the executable frontier; completed, merged, and pruned nodes remain in the tree as evidence for future proposal and selection. Each node carries the fields listed in Table 8.

The idea tree is serialized to JSON after every controlled mutation and serves as the sole authoritative record of research state accumulated during a run. No hypothesis, score, artifact reference, or lesson exists outside this structure. The same state is also rendered to Markdown for dashboards and human inspection. The JSON representation is consumed by tools, while the Markdown view provides a readable account of the research process. Since every mutation is persisted immediately, the run can recover from crashes, agent restarts, and context-window compression without relying on conversational memory.

Agents do not edit the raw JSON file directly. The coordinator accesses the tree through controlled read views and mutation tools: compact frontier views, single-node views, pending-leaf views, and the constraint view used before ideation; plus mutations for adding nodes, updating lifecycle fields, pruning subtrees, setting metadata, and propagating insights. Executors receive only the assigned hypothesis, the research contract, and ancestor insights. This access pattern keeps the tree as a shared state object rather than a low-level log of tool traces.

We initialize from the official tuned Muon optimizer setting provided by the benchmark authors. This configuration applies Muon to transformer block weights and AdamW to embeddings, the output projection, and scalar parameters (biases and gains). With these tuned hyperparameters, the baseline reaches a FineWeb validation loss of $\leq 3.28$ in 3,325 steps. Selecting a well-tuned official baseline as the starting material is intentional: it places the agent in a regime where naive hyperparameter sweeps are unlikely to yield large gains, thereby testing the agent’s ability to identify and implement genuinely novel optimizer improvements rather than harvesting easy wins from an undertuned starting point.

Agents iterate with the development evaluator throughout the search loop. After the search completes, the selected optimizer is re-evaluated with two held-out random seeds, and the reported test score is the average step count across those runs.

The evaluator runs the benchmark’s official evaluation script run_eval.py. The script launches the training job, monitors validation loss, and terminates as soon as val_loss $\leq 3.28$ is first reached. The score is the step count at termination, with lower being better. If the target is never reached, a penalty score exceeding 7,000 is assigned. All evaluation runs are executed on four NVIDIA A100-80 GB GPUs using the official benchmark evaluation code.

The agent is initialized with the following task description.

The initial material is the default autoresearch repository. The main artifact is train.py, a single-GPU decoder-only Transformer pretraining script derived from nanochat. Data preparation, tokenization, validation-token selection, and evaluation helpers live outside the editable artifact, primarily in prepare.py. The development evaluator uses the fixed validation shard provided by the benchmark. The default model is a roughly 50M-parameter Transformer, and the baseline reports the final validation bits-per-byte after the fixed training run.

During search, agents optimize only against the development evaluator, which runs the training script on the benchmark’s fixed prepared data and reports the final validation loss. The held-out score reported in the main experiments is obtained by rerunning the selected train.py with two held-out random seeds and averaging the resulting final losses. This separates the fast, single-run feedback used for hypothesis exploration from the seed-averaged score used for final comparison.

The evaluator executes uv run train.py from the benchmark repository. At the end of training the script prints a structured summary including val_bpb, peak_vram_mb, training_seconds, num_steps, and mfu_percent. The primary score is the final val_bpb, with lower values better. The training script enforces a 300-second training budget; runs that crash, time out, fail to fit in memory, or do not emit a parseable final val_bpb are treated as failed experiments. The agent may use secondary diagnostics such as step count and peak memory to interpret failures, but merge decisions are based on the loss metric.

The agent is initialized with the following task description.

Prior harness-engineering studies may begin from codebase variants that differ in interface extensibility, tool set, or prompt structure, creating comparison confounds. To ensure a fair and reproducible starting point we use the official terminal-agent codebase (terminus-2) distributed with the benchmark verbatim, without any modifications and without introducing additional interfaces. The backbone model is GPT-5.5. The agent operates as a ReAct-style loop that issues tmux keystrokes to a persistent terminal session.

We stratify the 89 tasks by difficulty and sample a 36-task development set and a 53-task held-out test set. The stratification ensures that both splits have balanced difficulty coverage, preventing a scenario where the agent is evaluated on an atypically easy or hard subset. Agents iterate exclusively on the development set during the research loop. The test set is evaluated only once after the search completes, to report held-out performance.

The evaluator is the official Harbor evaluation harness distributed with Terminal-Bench 2.0. Both development and test evaluations use 8 concurrent workers. The evaluator reports pass rate as the fraction of tasks solved correctly, with higher being better. The baseline achieves 58.33% on the development set (21 out of 36 tasks).

The agent is initialized with the following task description.

The initial material is a simple ReAct-based search harness centered on single_agent_gpt.py. For each question, a GPTAgent runs a single ReAct trajectory with a system prompt, issues web-search and page-visit tool calls, and returns a short final answer. The online BrowseComp setting uses SearchTool and VisitTool; the local-corpus tool variants are available in the codebase but are not the primary path for this benchmark. The baseline uses the simple-evals BrowseComp query template and grader template, with the same model family used for the answering agent and the grader in the reported runs.

The development split contains 50 BrowseComp questions and is used for all iterative optimization. The held-out test split contains 300 non-overlapping BrowseComp questions and is reserved for final verification. In the released harness, these splits are exposed as data/bc_val.jsonl and data/bc_test.jsonl. Agents may inspect and evaluate on the development split during the search loop, but they may not edit either data file or use the test split for iteration.

The development evaluator is uv run python run_eval.py --data data/bc_val.jsonl --workers 8 --run-name {node_id}. The held-out evaluator uses the same entry point with data/bc_test.jsonl and a distinct run name. The evaluator sends each question to the harness, collects the final answer, normalizes it through the fixed BrowseComp answer-grading path, and reports accuracy as correct/total. Items with repeated execution errors or unparsable final answers are counted as errors and do not contribute to the correct count. The primary metric is accuracy, with higher values better.

The harness exposes two browsing tools to the answering agent. SearchTool accepts one or more search queries and returns compact web search results, while VisitTool fetches and cleans page content for a selected URL. The tool names and argument schemas are fixed so that alternative harness designs remain compatible with the same ReAct agent interface. Agents may change the system prompt, rollout orchestration, response parsing, candidate aggregation, and other harness code, but the evaluation script, question files, grader template, and reference answers are fixed.

After optimization on BrowseComp, the resulting search harness is frozen and evaluated directly on unseen search-agent tasks, including HLE and DeepSearchQA, without additional task-specific optimization. The same code path, search/visit tool interface, and answer-production protocol are reused. This transfer test checks whether the discovered harness changes improve general browsing behavior rather than merely fitting the BrowseComp development questions.

The agent is initialized with the following task description.

The initial material is the unmodified default data-synthesis pipeline distributed with the benchmark. The baseline pipeline consists of five stages: extracting candidate facts from a topic seed, rating facts by information value, constructing a question whose answer is one selected fact, obfuscating the topic with supporting constraints, and applying structural verification. The generator, solver, and judge use GPT-5.5 through an OpenAI-compatible interface.

The benchmark provides frozen seed snapshots for both evaluation splits. The development split contains 50 seeds and is used exclusively for iterative optimization. The held-out test split contains 100 seeds and is reserved for final verification after the search loop completes. The pipeline generates at most one QA item per seed. Fixing the seed snapshots makes the benchmark self-contained and ensures that measured improvements reflect changes to the synthesis mechanism rather than changes to the data distribution.

The evaluator is the benchmark’s fixed run_eval.py entry point. It loads the requested seed split, runs the synthesis pipeline, solves each generated item four times with a web-search-enabled ReAct model, and reports a JSON summary on the final line of standard output. The primary metric is:

where $\mathrm{pass@1}$ is $1$ if the first attempt is judged correct, and $\mathrm{pass@4}$ is $1$ if at least one of four attempts is correct. Higher scores indicate questions that remain solvable in principle but are not answered reliably on the first attempt.

The agent is initialized with the following task description.

The initial material is the unmodified default synthesis pipeline distributed with the benchmark. The pipeline takes each seed, generates several candidate contest problems, filters candidates for answer format, rationale consistency, held-out overlap, and near-duplicate overlap, and evaluates surviving candidates with a fixed solver. The default generator uses GPT-5.5-mini and the reference solver uses GPT-5.5. The answer policy is integer_0_999.

The benchmark uses locked seed files for both evaluation splits. The development split contains 10 seeds with 5 generated candidates per seed, yielding up to 50 candidates total. The held-out test split contains 12 seeds with 8 generated candidates per seed, yielding up to 96 candidates total. Each seed specifies a mathematical topic, technique, and target difficulty. Agents iterate exclusively on the development split during the research loop. the test split is reserved for milestone and final verification.

The evaluator is the fixed run_eval.py entry point. It loads the requested split, runs the synthesis pipeline, applies validity and novelty filters, evaluates each surviving candidate with the fixed solver, and prints a JSON metrics object on the final line of standard output. The primary metric is:

where $N$ is the number of generated candidates before filtering, $\mathrm{pass@1}$ indicates whether the first solver sample is correct, and $\mathrm{pass@4}$ indicates whether at least one of four solver samples is correct. Filtered candidates are treated as zero. This metric rewards problems that are solvable under repeated attempts but not immediately solved on the first sample.

The agent is initialized with the following task description.