Reading + AI

Auto-Research Arena: Benchmarking and Advancing Auto-Research in Language Modeling

Xi Chen, Chongjian Ge, Yiran Xu, Yiqun Mei, Henrik I. Christensen, Hao Tan, Ziwen Chen, Hanwen Jiang

arXiv:2610.35866v2 · This workspace stays on this version. Check arXiv for newer versions

Affiliations and author notes
  • University of California San Diego
  • Adobe Research
Some source details need review 7 to check

Original formulas, table values and figures are preserved. Comparison can miss differences; use the original PDF to check details.

Reading view checked: 71 math expressions, 1 rendering failures; 19 of 20 images available.

Compared 128 of 128 source passages against 28 of 28 PDF pages.

  • Some figure, table or graphic images are unavailable; the original paper is still accessible. (1)
  • Some formulas could not be rendered; their original TeX is shown. Check the original PDF. (1)
  • A3_SS1_tab1: blank, oversized or incomplete source rendering; capture unavailable.

The PDF visibly reads “MLRC-Bench (Zhang et al., 2025c),” while the HTML reads “MLRC-Bench ([Zhang et al., 2025a](#bib.bib11)).” Check PDF page 3

HTML A3_SS1_p2_1 says “Table [C.1](#A3.SS1) gives mechanisms, objectives, and comparisons with prior work.” PDF p.21 says “Table 7 gives mechanisms, objectives, and comparisons with prior work.” The original PDF supports Table 7, contradicting the HTML reference. Check PDF page 21

HTML S4_SS3_p3_1 says “Appendix  details how these ideas are selected and transformed.”; PDF page 8 says “Appendix C.3 details how these ideas are selected and transformed.” Check PDF page 8

PDF page 9 reads “ones (Tab. 8).”, contradicting HTML block S4_SS4_p3_1, which reads “ones (Tab. ).” Check PDF page 9

19 of 20 figure, table and graphic images captured. Open original PDF

Abstract

As autonomous research systems become more capable, benchmarks must both evaluate their performance and support meaningful research progress. We introduce Auto-Research Arena, a multi-target benchmark for evaluating autonomous research systems and discovering ideas in language modeling. Arena supports diverse research objectives within a shared experimental environment by separating admissibility from improvement: model quality and resource constraints determine whether a result qualifies, while the selected target measures progress among admissible results. We instantiate Arena in language model pretraining with nine targets spanning model, data, training, and inference efficiency, and evaluate seven research harnesses using two foundation models. Arena clearly differentiates existing harnesses across research objectives, with several sophisticated methods outperformed by our simple sequential search baseline, indicating substantial headroom in current auto-research systems. Beyond evaluation, Arena reveals a substantially broader frontier of model and system ideas aligned with active research directions. Reusing these discoveries in a new quality-driven search yields lower training loss under matched data and FLOPs and lower held-out BPB at the same model size. Together, Arena provides a common framework for benchmarking autonomous research and expanding its scope in language modeling. The code is available at https://github.com/Auto-Research-Arena/auto-research-arena.

Figure 1
Figure 1: Auto-Research Arena evaluates research systems and expands the idea frontier. Left: Arena differentiates research harnesses across nine complementary objectives. Our simple Sequential Search baseline remains competitive, whereas the TPE method tailored to validation-loss minimization underperforms in this broader evaluation. Right: Multi-target research reveals a broad frontier of ideas. A program combining these discoveries achieves lower training loss than the reference program under matched data and FLOP budgets, as well as lower final held-out BPB at the same model size.

Advances in LLMs and research harnesses have enabled increasingly capable autonomous research systems (Tang et al., 2025; Liu et al., 2026). These systems iteratively propose ideas, conduct experiments, and use the results to guide further research. Auto-research benchmarks provide a common basis for comparing these systems, but their role extends beyond evaluation. By defining clear objectives and constraints, a well-designed benchmark helps researchers identify, validate, and build on useful improvements. However, existing benchmarks remain limited both in evaluating research systems and in supporting broad research ideas (Chan et al., 2025; Wijk et al., 2025; Nathani et al., 2025; Karpathy, 2026; Intology, 2026).

To address this gap, we introduce Auto-Research Arena, a comprehensive benchmark for evaluating auto-research systems and advancing idea discovery in language modeling. Departing from prior benchmarks that define progress primarily through a particular objective within each research problem, Arena defines multiple research objectives within a common experimental environment. These objectives address complementary bottlenecks across the model lifecycle, including model size, data consumption, training computation and memory, and inference memory and latency. The shared environment provides a common experimental basis for evaluating whether a harness remains effective across different research objectives. Meanwhile, different objectives reward different types of improvements. For example, reducing model size or inference memory can count as progress even when prediction loss remains unchanged. By recognizing these complementary forms of progress, Arena supports exploration of a broader range of model designs and systems optimizations.

To support this broad range of research objectives within a unified framework, Arena specifies both what to improve and which requirements a candidate must satisfy. That is, Arena separates the standard for admissibility from the direction of improvement. Each objective is formulated as a constrained optimization problem. Constraints on model quality and relevant non-target resources determine whether a candidate is admissible. Among admissible candidates, a reduction in the selected resource metric constitutes improvement. Changing the target metric defines a different research objective within the same framework. This formulation supports consistent evaluation across diverse objectives, while excluding gains that violate the specified quality or resource constraints.

Beyond evaluating objectives independently, Arena supports research across objectives. Separate searches can pursue different objectives in parallel, each exploring improvements relevant to its target. Compatible improvements discovered in one search can then be incorporated into the starting program for a subsequent search targeting a different objective. This enables both parallel exploration of distinct research directions and cumulative progress through the reuse of earlier discoveries. By explicitly specifying objectives, constraints, budgets, and starting programs, Arena makes it possible to measure how prior discoveries affect subsequent research.

We evaluate six LLM-based auto-research methods and a Tree-structured Parzen Estimator baseline (Ferreira et al., 2026) across Arena’s nine objectives. Arena provides a discriminative and meaningful evaluation of research harnesses: methods exhibit clear differences across objectives, with no single harness dominating the benchmark. Two results further support the evaluation. TPE, specialized for the original validation loss-oriented setting, performs worst in Arena’s broader multi-target evaluation. Moreover, our simple Sequential Search baseline ranks second overall with both foundation models, outperforming several substantially more sophisticated harnesses and suggesting large headroom in current auto-research system design. The recorded search trajectories help explain how these final outcomes arise. Higher-performing harnesses tend to sustain progress throughout the search budget, whereas lower-performing methods often plateau earlier or improve less consistently.

Multi-target research also exposes a substantially broader idea frontier. We empirically find that different objectives lead to qualitatively different discoveries, yielding 32 findings across nine idea families spanning model architecture, optimization, memory, and inference systems. Many of these discoveries align with active directions in LLM research. For example, when optimizing parameter efficiency, agents discover looped Transformers through weight sharing. They further learn depth-specific residual gains to adapt the residual updates across repeated applications of the shared block. This finding closely parallels recent work identifying residual adaptation as a key ingredient of looped Transformers (Wang et al., 2026). Similar connections appear across other discoveries, although alignment with existing research does not imply identical solutions. Several discoveries follow design principles similar to those of prior work but implement them through distinct mechanisms, offering alternative approaches to the same research problems. Together, these findings show how Arena’s diverse objectives expand the range of ideas explored by autonomous research systems.

Finally, we examine whether ideas discovered through Arena’s lightweight searches remain useful beyond the search setting. The original searches train each candidate for only 10 minutes. We provide the collected discoveries to a new search focused on model quality, which can select, combine, and modify these ideas. We then train the resulting model for one hour from random initialization. Unlike the original searches, which optimize individual efficiency targets, this experiment evaluates improvements in model quality directly. Relative to the reference run, training loss decreases from 2.675 to 2.544 at matched training data and from 2.617 to 2.544 at matched cumulative FLOPs. Held-out BPB also decreases from 0.9295 to 0.9112 at essentially the same model size. These results show that discoveries from Arena’s small-scale, multi-target searches can improve model quality through combination and adaptation. Arena thus supports both the evaluation of autonomous research systems and the discovery of reusable research ideas.

Autonomous research benchmarks. PaperBench (Starace et al., 2025) evaluates replication of published research, while EXP-Bench (Kon et al., 2026) assesses the design and execution of paper-derived experiments. These benchmarks emphasize implementing and validating specified research content. More open-ended settings include MLE-bench (Chan et al., 2025) and RE-Bench (Wijk et al., 2025), which evaluate ML engineering and research, alongside MLGym (Nathani et al., 2025), MLRC-Bench (Zhang et al., 2025a), and AstaBench (Bragg et al., 2026), which support standardized experimentation and evaluation across ML and scientific tasks. Within language modeling, Karpathy’s autoresearch (Karpathy, 2026) minimizes validation loss under a fixed training-time budget, whereas NanoGPT Bench (Intology, 2026) targets faster training at a required model quality. Related benchmarks evaluate reproduction of NanoGPT improvements (Zhao et al., 2025) and correctness-constrained computational speedups (Ouyang et al., 2025; Press et al., 2025; Shetty et al., 2025). However, existing benchmarks provide limited evaluation of auto-research systems across complementary objectives. Arena instead unifies diverse objectives within a shared experimental setting, revealing how the research objective shapes both harness performance and the resulting idea frontier.

Autonomous research systems. Complementing these benchmarks, research harnesses determine how agents organize experiments and use accumulated evidence. End-to-end systems connect ideation, experimentation, and reporting (Lu et al., 2026; Tang et al., 2025), while AIDE (Jiang et al., 2025) and AIRA (Toledo et al., 2025) organize iterative code improvement through search. AutoScientists (Gao et al., 2026) coordinates decentralized teams, Arbor (Jin et al., 2026) maintains a persistent hypothesis tree, and GEAR (Jeddi et al., 2026) evolves populations of research states. Heuresis (Antoniades et al., 2026) further compares search strategies across quality, diversity, and novelty. Arena evaluates how the relative strengths of research harnesses vary across objectives.

Auto-Research Arena defines a family of research problems that optimize different resource targets within a shared language-modeling environment. Fig. 2 illustrates the research loop, from selecting a target and evaluating candidate programs to identifying each harness’s best admissible program and comparing performance across targets. In the following, we present the general formulation in Sec. 3.1 and its language-modeling instantiation in Sec. 3.2.

Consider a testbed with a fixed data pool, evaluator, and execution environment. An auto-research harness searches over executable programs p∈Pp\in\mathcal{P}, where candidate programs may modify the model architecture, optimizer, training schedule, and inference implementation.

For each objective jj, Arena minimizes a resource metric Target⁡j(p)\operatorname{Target}_{j}(p) subject to a quality threshold τ\tau (shared by all targets jj), target-specific resource limits bj\mathbf{b}_{j}, and correctness checks:

minimize⁡p∈PTarget⁡j(p)subject to⁡L(p)<τ,Resources⁡j(p)≤bj.\begin{aligned}\displaystyle\underset{p\in\mathcal{P}}{\operatorname{minimize}} & \displaystyle\operatorname{Target}_{j}(p) \\ \displaystyle\operatorname{subject~to} & \displaystyle\mathcal{L}(p)<\tau, \\ & \displaystyle\operatorname{Resources}_{j}(p)\leq\mathbf{b}_{j}.\end{aligned}
(1)

Here, L(p)\mathcal{L}(p) denotes validation loss, for which lower values are better. The vector Resources⁡j(p)\operatorname{Resources}_{j}(p) contains the constrained resource measurements, and bj\mathbf{b}_{j} specifies their limits. Resource inequalities apply componentwise. This formulation separates admissibility from improvement: candidates qualify by satisfying the quality, resource, and correctness requirements, while lower target values indicate better outcomes among qualifying candidates.

Efficiency targets. Arena instantiates this formulation with nine objectives spanning model, training, and inference efficiency. These objectives address complementary bottlenecks in model development and deployment. Tab. 1 summarizes their measurements and practical motivations.

Each metric captures a distinct dimension of efficiency. Parameter efficiency measures the model size needed to achieve the required quality. Training efficiency measures the resources needed to reach that quality, while inference efficiency measures the resources needed to use the trained model. Optimizing these metrics separately defines distinct research problems within the same experimental environment.

Figure 2
Figure 2: Overview of Auto-Research Arena Pipeline. Each search run optimizes one of nine efficiency targets. A research harness proposes and implements changes to a reference program. Arena evaluates the resulting candidates, records outcomes, and provides feedback. The best admissible candidate determines the benchmark result.

We instantiate Arena using Karpathy’s autoresearch language-modeling testbed (Karpathy, 2026), in which candidate solutions are compact LLM pretraining programs. We retain its fixed training-data pool and lightweight experimental setup, replacing loss minimization with the constrained formulation in Eq. 1. Each track optimizes one of nine efficiency metrics subject to requirements on model quality and relevant non-target resources.

Model quality. Following the original testbed (Karpathy, 2026), we measure language-modeling quality using validation bits per byte (BPB). A candidate is admissible only if its validation BPB is below 1.051.05. This threshold is slightly above the reference program’s BPB, providing flexibility for structural changes while limiting quality degradation.

Resource constraints. Each track limits relevant non-target resources to control the trade-offs permitted during optimization. These constraints cover quantities such as model size, training FLOPs, peak training memory, and available training data. Inference tracks also constrain other serving costs. For example, the live-state-memory track imposes a latency bound, while the decode-latency track imposes a live-state-memory bound. Each track therefore measures improvement in its selected metric while requiring other resource costs to remain within specified limits.

Measurement. The Arena evaluator measures all target metrics and constrained resources directly. Depending on the target, measurements are taken after a fixed 600-second training period or when the candidate first meets the quality requirement. Training metrics include consumed tokens, FLOPs per token, peak allocated GPU memory, elapsed training time, and optimizer updates. Inference metrics are measured after training using standardized prompt and generation workloads. Appendix A provides complete metric definitions and resource constraints for all nine tracks.

Figure
Table 1
TargetMeasured quantityWhy it matters
Model
Total parametersNumber of model parametersModel storage and weight-memory footprint (Lan et al., 2019).
Training
Data consumptionTokens used to reach target qualityLearning efficiency under a token budget (Hoffmann et al., 2022).
Optimizer updatesUpdates to reach the quality thresholdConvergence in fewer updates (Gupta et al., 2018).
Training FLOPsFLOPs per training tokenCompute demand per token (Hoffmann et al., 2022).
Peak memoryPeak allocated GPU memoryTraining feasibility on memory limits (Chen et al., 2016).
Training timeTime to reach the quality thresholdTraining turnaround (Mattson et al., 2020).
Inference
Live-state memoryMemory per active requestConcurrent serving capacity (Kwon et al., 2023).
Decode latencySequential decoding latencyAutoregressive generation latency (Shazeer, 2019).
Long-prompt latencyPrefill + decoding latencyLong-context response latency (Zhong et al., 2024).
Table 1: Efficiency targets in Auto-Research Arena. Each track minimizes one target subject to a shared model-quality threshold and target-specific resource constraints.
View original tableOriginal Table 1

Example: parameter efficiency. The parameter-efficiency track seeks the smallest model that reaches the required quality within the specified training-compute and memory budgets. The remaining eight tracks follow the same formulation, each with its own target metric and corresponding resource constraints.

We evaluate Arena along its two central roles: benchmarking autonomous research systems and supporting broad idea discovery. We then use its controlled multi-target setting to study how research evolves under different objectives, constraints, budgets, and research histories.

Benchmark–harness interface. Arena provides the environment, task specification, reference program, evaluator, and experiment budget, while each harness determines how to conduct the search. Starting from the reference program, a harness proposes code modifications, submits candidates for evaluation, observes the measured outcomes, and decides what to explore next. Arena records every experiment and independently reports the best recorded candidate satisfying the quality, resource, and correctness requirements, regardless of which program the harness ultimately retains. Each search receives 100 charged experiments; an experiment is charged once GPU execution begins, including runs that subsequently fail or are abandoned. Appendix B provides the full interface and protocol.

Reference baselines. We introduce two simple reference harnesses that differ primarily in how many research directions they retain. Sequential Search maintains one program and evaluates one modification at a time, adopting an admissible improvement before continuing. Beam Search retains up to four programs and evaluates two modifications of each before selecting the next frontier. Both condition subsequent proposals on the accumulated experimental history, including unsuccessful attempts. Appendix B provides their detailed proposal, selection, and stopping policies.

Tasks and resources. We evaluate all nine Arena objectives defined in Sec. 3, each with target-specific resource constraints and the shared quality requirement BPB <1.05<1.05. Each candidate runs on one NVIDIA A100 80GB GPU with seed 42 and up to 600 seconds of training, excluding compilation, validation, and the first ten warmup updates. All searches use the same approximately 631M-token ClimbMix (Diao et al., 2025) training pool and held-out BPB evaluator; training-token consumption includes repeated exposure to the data.

Models and harnesses. We evaluate six LLM-based research harnesses with Claude Opus 5 (Anthropic, 2026) and GPT-6 Astra (OpenAI, 2026), both using medium reasoning effort. In addition to Sequential Search and Beam Search, we evaluate AutoScientists (Gao et al., 2026), Arbor (Jin et al., 2026), GEAR (Jeddi et al., 2026), and Heuresis (Antoniades et al., 2026), which respectively use multi-agent coordination, hypothesis trees, evolutionary search, and diversity-preserving archives. We additionally include TPE (Ferreira et al., 2026), a non-agentic baseline restricted to a predefined hyperparameter space, in the Opus 5 experiments.

Scoring. Each objective is scored by the best recorded admissible target value, with lower values better. For training-token consumption, optimizer updates, and training time, we use the first 30-second validation checkpoint satisfying BPB <1.05<1.05. We summarize performance across objectives with different units and scales using Elo ratings from pairwise harness comparisons within each objective. Appendix  gives the scoring procedure.

Tab. 2 reports the best results across Arena’s nine objectives, while Fig. 3 shows representative search trajectories.

Arena provides a discriminative evaluation. Research harnesses exhibit substantial differences across Arena’s objectives, with no single method dominating the benchmark. With Opus 5, AutoScientists achieves the highest aggregate Elo, followed by Sequential Search and Arbor; with GPT-6, GEAR ranks first, followed by Sequential Search. The variation across objectives shows that Arena distinguishes complementary strengths of research systems rather than reducing their performance to a single research setting.

Simple search is surprisingly competitive. Despite its minimal design, Sequential Search ranks second overall with both foundation LLMs, outperforming several substantially more sophisticated harnesses. Its strong performance suggests substantial headroom in current auto-research system design: more elaborate orchestration does not yet consistently translate into stronger performance.

Specialization does not necessarily transfer across research problems. TPE provides a complementary sanity check. Designed for Karpathy’s autoresearch setting (Ferreira et al., 2026), TPE performs substantially worse than all LLM-based harnesses across Arena’s broader set of objectives. Strong performance in a specialized research setting therefore need not translate into broad research capability, illustrating the value of evaluating research systems across complementary objectives.

Table 2
Table 2: Benchmark performance across nine research objectives. Lower objective values and higher Elo ratings are better; bold indicates the best result within each foundation-model group. Charged experiments reports the mean number of budgeted experiments across objectives. Elo is computed separately within each foundation-model group. TPE is a non-agentic baseline and is included only in the Opus 5 comparison.
Figure 3
Figure 3: Research trajectories behind the Opus 5 benchmark results. Curves track the best admissible result found as each search proceeds, ordered by charged experiments; lower is better. Counts include the reference evaluation and charged failures, and dots mark the end of each recorded search. Training time is measured at 30-second validation checkpoints, and KV-cache allocation is shown on a logarithmic scale.

Research trajectories support the benchmark scores. The endpoint differences in Tab. 2 are also reflected in the search trajectories in Fig. 3. Stronger harnesses typically make sustained progress over the experiment budget, whereas weaker methods plateau earlier or improve less consistently. For example, TPE stops improving relatively early in the parameter search, while GEAR and AutoScientists continue to find improvements near the end of their recorded searches. These trajectories provide complementary evidence that the benchmark scores reflect systematic differences in research progress rather than isolated final outcomes.

Similar outcomes can hide different research efficiency. Endpoint performance does not fully characterize how a result is reached. On the Opus 5 training-time objective, GEAR, Arbor, Beam Search, and Heuresis all reach 270 seconds, but first attain this value after 12, 39, 50, and 99 charged experiments, respectively. Thus, harnesses with identical final scores can differ substantially in how efficiently they discover the same level of improvement.

Multi-target research substantially broadens the idea frontier. Beyond routine hyperparameter tuning, Karpathy’s reported loss-driven search identifies two architectural changes, smear and backout  (Karpathy, 2026). Across Arena’s efficiency objectives, our search catalogue contains 32 distinct findings, including 20 that were accepted as improvements during search, which we consolidate into nine idea families spanning model architecture, training, and inference. Different objectives expose qualitatively different research directions, from architectural changes under model-efficiency targets to memory and systems optimizations under training and inference targets.

Table 3
Discovered idea: Loop Transformer with depth-specific residual gain Optimization target: Parameter efficiency
BEFORE: INDEPENDENT BLOCKS
for ℓ=0,…,7:u←h+Attn⁡ℓ(h)h←u+MLP⁡ℓ(u)\begin{aligned} &\mathbf{for}\ \ell=0,\ldots,7:\\ &\quad u\leftarrow h+\operatorname{Attn}_{\ell}(h)\\ &\quad h\leftarrow u+\operatorname{MLP}_{\ell}(u)\end{aligned}
[0pt][0pt] AFTER: SHARED BLOCKS, DEPTH-AWARE GAINS
[0pt][0pt]  \begin{aligned} &\mathbf{for}\ \ell=0,\ldots,7:\\ &\quad u\leftarrow h+{\color[rgb]{0.0977,0.3672,0.5313}\alpha_{\ell}\operatorname{Attn}_{\ell\bmod B}(h)}\\ &\quad h\leftarrow u+{\color[rgb]{0.0977,0.3672,0.5313}\beta_{\ell}\operatorname{MLP}_{\ell\bmod B}(u)}\end{aligned}
B=1B=1: one stored block, eight applications. The learned αℓ,βℓ\alpha_{\ell},\beta_{\ell} control attention and MLP residual updates separately at each depth.
RELATION TO PRIOR WORK Recent work scales residual updates with loop count (Wang et al., 2026); here the gains are learned separately for each depth.
Table 3: A representative searched idea by improving parameter efficiency. Blue highlights searched changes; more discoveries are shown in Appendix C.
View original tableOriginal Table 3

The discovered idea frontier mirrors active LLM research. The breadth of Arena’s discoveries is not merely a larger collection of benchmark-specific optimizations: many of them reach questions central to current LLM research. A representative example emerges from parameter efficiency (Tab. 3). Agents first discover recurrent weight sharing, effectively turning the model into a looped Transformer, and further identify the need to adapt residual updates across repeated depth, learning separate attention and MLP residual gains at each depth. This closely mirrors recent work studying residual adaptation as a key component of looped Transformers (Wang et al., 2026). Similar correspondences appear across attention-state reduction through cross-layer K/V sharing (Brandon et al., 2024), MQA/MLA (Shazeer, 2019; DeepSeek-AI, 2024), conditional memory (Cheng et al., 2026), optimizer-state reduction (Zhang et al., 2025b), activation-memory efficiency (Korthikanti et al., 2022), and inference optimization (Aminabadi et al., 2022). Importantly, these discoveries are not simply reproductions of existing methods: they often share the same underlying principles while arriving at different technical mechanisms, offering alternative perspectives on active research questions. The resulting idea frontier therefore demonstrates a broader value of Arena: its diverse objectives expand not only what research systems are evaluated on, but also the space of meaningful research directions they explore.

The broader idea frontier accompanies substantial measured progress. The searches producing these discoveries also achieve large admissible gains across distinct resource dimensions. In the Opus 5 group, the best programs reduce model size from 50.33M to 13.20M parameters, training computation from 239.08M to 52.15M FLOPs per token, and peak training memory from 43.96 to 0.373 GiB, while satisfying BPB <1.05<1.05 and the corresponding resource constraints. However, these endpoint programs combine multiple changes accumulated during search, so their gains do not establish whether the discovered ideas themselves provide reusable knowledge beyond the searches that found them. We test this directly in Sec. 4.3.

Figure 4
Figure 4: Discovered programs can transfer beyond the search horizon. Programs discovered with 10-minute candidate training are retrained from random initialization for 60 minutes alongside the reference. Several retain their advantages for substantially longer than the original search horizon, although these gains can diminish later as horizon-specific training policies take effect. Curves show held-out BPB against training time. Annotated comparisons use checkpoints after the learning-rate schedule reaches its floor, except in figure (c), where AutoScientists’ final BPB defines the shared target.

Arena discovers mechanisms under specific objectives and short experimental budgets. To assess their usefulness beyond these conditions, we first test whether a library of discovered ideas can guide a new search for better model quality. We then ask a stricter question: can winning programs from the original Arena searches retain their advantages when retrained for longer without further search?

Combining discovered ideas improves model quality. We collect discoveries across Arena’s objectives and harnesses into an idea library to guide a quality-driven search. The agent can select, modify, and combine mechanisms from different research trajectories rather than continue from a single winning program. We extend the training time budget to 1 hour in this search.

The resulting design improves model quality under both matched-budget evaluations (Fig. 1). At matched training data, it reduces training loss from 2.675 to 2.544 (4.9%); at matched training FLOPs, it reduces training loss from 2.617 to 2.544 (2.8%). Final held-out BPB also decreases from 0.9295 to 0.9112 at the same model size (50.33M parameters). The resulting design selectively adapts and simplifies ideas rather than merely combining their original implementations. These results demonstrate that efficiency-targeted discoveries can serve as reusable model-design knowledge for improving quality. Appendix  details how these ideas are selected and transformed.

Discovered programs can transfer far beyond the search horizon. We next ask a stricter question: can a program found under a short training budget be scaled directly, without further search? Arena normally evaluates candidate programs after 10 minutes of training. We select two programs from each of the training-time, token, and update objectives and retrain them from random initialization for 60 minutes alongside the reference program.

Several discoveries retain their advantages far beyond the horizon at which they were found. A token-optimized GEAR program reaches 0.9593 BPB after 30 minutes using 1.004B tokens, while the reference requires approximately 47.6 minutes and 1.771B tokens to reach the same quality, corresponding to 43.3% fewer tokens. Its quality advantage persists until approximately 51 minutes, extending to five times the original search horizon. Similarly, a training-time program discovered by AutoScientists outperforms the reference over the 15–46-minute interval and reaches 0.9647 BPB after 30 minutes, a quality level that the reference reaches only after approximately 45.8 minutes, corresponding to 34.5% less training time (Figure 4; Table 5). Reference costs at matched quality are estimated by linear interpolation.

The transfer is substantial but not indefinite. The GEAR program’s learning rate reaches its 2% floor around 27 minutes, while the AutoScientists schedule reaches zero around 24 minutes. Their early advantages consequently diminish as training continues; by 60 minutes, for example, the reference reaches 0.9295 BPB compared with 0.9478 for the GEAR program. Thus, short-horizon search can discover programs whose advantages persist for four to five times the original search horizon, while horizon-dependent training policies can ultimately limit direct extrapolation to longer training.

Arena also enables controlled study of what shapes the auto-research process. We examine quality constraints, budget allocation between evaluation depth and search breadth, and the role of research history in cross-target search.

Constraints shape the research space. We vary the BPB threshold across 1.021.02, 1.051.05, and 2.02.0 for parameter- and memory-efficiency searches with Sequential Search and Beam Search (Tab. 5). Relaxing the threshold from 1.051.05 to 2.02.0 allows Sequential Search to reduce parameters from 18.94M to 0.69M, while its separate peak-memory search reduces memory from 0.83 GiB to 0.035 GiB. The additional quality slack permits substantially more aggressive efficiency improvements. Conversely, tighter thresholds restrict which structural changes remain admissible. Quality constraints thus shape not only the final results, but also which regions of the design space can be developed.

Budget allocation creates a depth–breadth trade-off. For parameter minimization under BPB <2.0<2.0, we vary candidate training time from 1,200 to 6,000 seconds under a shared 20 GPU-hour training budget, trading fewer, longer evaluations for more, shorter ones (Tab. ). The shortest setting evaluates 60 candidates and finds the smallest model; longer settings yield better BPB but more parameters. These outcomes illustrate the trade-off between evaluation depth and search breadth, with different allocations favoring different quality–efficiency balances.

Paper table
ObjectiveHarnessResource savingQuality advantage to60-min BPB
Reference––0.9295
Training timeSeq. Search28.5%∼\sim44 min0.9662
AutoScientists34.5%∼\sim46 min0.9647
Training tokensGEAR43.3%∼\sim51 min0.9478
Seq. Search34.4%∼\sim42 min0.9765
Training updatesAutoScientists30.1%–0.9409
Seq. Search39.6%–0.9365
Paper table
HarnessBPB LimitParametersPeak GPU Memory
Params. (M)BPBMemory (GiB)BPB
Beam Search<1.02<1.0223.59341.01853.97531.0175
<1.05<1.0519.66101.04481.67071.0371
<2.0<2.00.46491.50300.16251.2312
Sequential Search<1.02<1.0226.47671.01956.88891.0162
<1.05<1.0518.93991.04970.83411.0402
<2.0<2.00.69241.99630.03481.7200
Table 4 Direct transfer beyond the search horizon. Programs discovered with 10-minute training are retrained for up to 60 minutes without further search. Resource savings compare each program’s 30-minute checkpoint with the reference at the same held-out BPB.
Table 5 Looser quality constraints enable more aggressive efficiency gains. Results use 30 completed runs per search; bold marks each harness’s best value.
View original tableOriginal table
Table 6
Table 6: Research can build across objectives. Each sequence continues from a program discovered under the first objective and optimizes a new objective. The second search can preserve the earlier gain or trade part of it for progress on the new objective. Δ2\Delta_{2} reports the change from the first-search program; negative values indicate improvements.

Cross-target research builds on prior discoveries. We initialize searches under new objectives from programs discovered under earlier objectives, transferring code while reinitializing model weights (Tab. 6). Subsequent searches can accumulate compatible gains: parameter savings are fully preserved while training FLOPs decrease by a further 33.7%, and a FLOP-optimized program preserves its FLOP count while uncached decode latency decreases by 87.5%. Cross-target continuation can also introduce trade-offs: optimizing training time from the parameter-optimized program reduces time to quality by 21.1% but increases parameters by 24.6%. Prior discoveries therefore provide foundations for both cumulative gains and new trade-offs across objectives.

Together, these studies show how constraints define admissible designs, budgets balance evaluation depth against search breadth, and research history supports cumulative gains or trade-offs across targets. Therefore, Arena enables systematic study of how autonomous research develops over time.

We introduced Auto-Research Arena, a multi-target benchmark for autonomous research across nine quality-constrained language-model efficiency objectives. Arena reveals complementary harness strengths and a broad frontier of architectural and systems ideas. Discoveries can seed searches under new objectives, while a program combining selected ideas can improve model quality. Controlled studies further show how constraints, budgets, and research history shape progress. Together, these results support evaluating autonomous research across complementary objectives and studying how useful discoveries accumulate.

Limitations. Our scope is explicitly limited to auto-research in language modeling, rather than the full end-to-end research process or scientific novelty. Idea-family groupings are interpretive, and noisy system measurements require care when interpreting small differences.

LLM-based research agents are the subjects of evaluation in this work; their roles in proposing ideas, implementing programs, and conducting experiments are described in the paper.

Separately, we used AI tools to assist with literature search, compilation of experimental records, text polishing for readability and figure preparation.

We have not used generative AI tools for helping develop theoretical models or conceptual frameworks, proposing or refining hypotheses, designing or providing feedback on research methodology or experiments, implementing methods, assisting with translation, supporting qualitative and thematic data analysis, and interpreting results.

Generating synthetic data sets, formulating mathematical claims, providing critical ingredients for proving mathematical claims, assisting in the writing of proofs, cleaning and reformating dataset, are not applicable to this work.

We have reviewed AI-assisted content against the relevant literature, source code, and experimental records. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

This work studies autonomous research systems in a controlled language-modeling environment. The experiments use existing language-model training data and do not involve human subjects, personal data collection, or deployment to users.

The evaluated agents can autonomously propose code changes and execute experiments, which may raise broader concerns about increasingly capable autonomous research systems. In our setting, their actions are restricted to a sandboxed experimental environment, a fixed program interface, predefined compute budgets, and standardized evaluation. We do not evaluate or enable autonomous interaction with external systems beyond the research environment.

We report both successful and unsuccessful research outcomes and distinguish measured improvements from interpretations of scientific novelty. We hope that the benchmark can support more systematic study of the capabilities and limitations of autonomous research systems.

References

  1. Aminabadi et al. (2022) R. Y. Aminabadi, S. Rajbhandari, M. Zhang, A. A. Awan, C. Li, D. Li, E. Zheng, J. Rasley, S. Smith, O. Ruwase, and Y. He DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale. arXiv preprint arXiv:2207.00032. Cited by: §4.2.
  2. Anthropic (2026) Anthropic Introducing claude opus 5. Note: https://www.anthropic.com/news/claude-opus-5Anthropic model release announcement Cited by: §4.Source
  3. Antoniades et al. (2026) A. Antoniades, D. Nathani, R. Saha, A. Amayuelas, I. Bercovich, Z. Weng, V. Baskaran, K. Bhatia, and W. Y. Wang Heuresis: search strategies for autonomous AI research agents across quality, diversity and novelty. arXiv preprint arXiv:2606.25198. Cited by: §2, Table 2, Table 2, §4.
  4. Bragg et al. (2026) J. Bragg, M. D’Arcy, N. Balepur, D. Bareket, B. Dalvi Mishra, S. Feldman, D. Haddad, J. Hwang, P. Jansen, V. Kishore, B. P. Majumder, A. Naik, S. Rahamimov, K. Richardson, A. Singh, H. Surana, A. Tiktinsky, R. Vasu, G. Wiener, C. Anastasiades, S. Candra, J. Dunkelberger, D. Emery, R. Evans, M. Hamada, R. Huff, R. Kinney, M. Latzke, J. Lochner, R. Lozano-Aguilera, N. Nguyen, S. Rao, A. Tanaka, B. Vlahos, P. Clark, D. Downey, Y. Goldberg, A. Sabharwal, and D. Weld AstaBench: rigorous benchmarking of AI agents with a scientific research suite. In International Conference on Learning Representations, pp. 110136–110223. Cited by: §2.
  5. Brandon et al. (2024) W. Brandon, M. Mishra, A. Nrusimha, R. Panda, and J. Ragan-Kelley Reducing Transformer Key-Value Cache Size with Cross-Layer Attention. arXiv preprint arXiv:2405.12981. Cited by: §4.2.
  6. Chan et al. (2025) J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, A. Madry, and L. Weng MLE-bench: evaluating machine learning agents on machine learning engineering. In The Thirteenth International Conference on Learning Representations, Cited by: §1, §2.
  7. Chen et al. (2016) T. Chen, B. Xu, C. Zhang, and C. Guestrin Training Deep Nets with Sublinear Memory Cost. arXiv preprint arXiv:1604.06174. Cited by: Table 1.
  8. Cheng et al. (2026) X. Cheng, R. Tian, W. Zeng, D. Dai, Q. Chen, B. Wang, Z. Xie, K. Huang, X. Yu, C. Deng, S. Zhou, C. Zhao, Z. Hao, Y. Li, H. Zhang, Z. Zhang, Y. Wei, M. Y. Xu, H. Zhang, D. Zhao, and W. Liang Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models. arXiv preprint arXiv:2601.07372. Cited by: §4.2.
  9. DeepSeek-AI (2024) DeepSeek-AI DeepSeek-V2: a strong, economical, and efficient mixture-of-experts language model. External Links: 2405.04434 Cited by: §4.2.
  10. Diao et al. (2025) S. Diao, Y. Yang, Y. Fu, X. Dong, D. Su, M. Kliegl, Z. Chen, P. Belcak, Y. Suhara, H. Yin, M. Patwary, Y. (. Lin, J. Kautz, and P. Molchanov Nemotron-CLIMB: clustering-based iterative data mixture bootstrapping for language model pre-training. In Advances in Neural Information Processing Systems, Vol. 38. Cited by: §4.
  11. Ferreira et al. (2026) F. Ferreira, L. Wobbe, A. Krishnakumar, F. Hutter, and A. Zela Can llms beat classical hyperparameter optimization algorithms? a study on autoresearch. External Links: 2603.24647 Cited by: §1, §4.1, Table 2, §4.
  12. Gao et al. (2026) S. Gao, A. Fang, and M. Zitnik AutoScientists: self-organizing agent teams for long-running scientific experimentation. arXiv preprint arXiv:2605.28655. Cited by: §2, Table 2, Table 2, §4.
  13. Gupta et al. (2018) V. Gupta, T. Koren, and Y. Singer Shampoo: preconditioned stochastic tensor optimization. In Proceedings of the 35th International Conference on Machine Learning, J. Dy and A. Krause (Eds.), Proceedings of Machine Learning Research, Vol. 80, pp. 1842–1850. Cited by: Table 1.
  14. Hoffmann et al. (2022) J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, O. Vinyals, J. Rae, and L. Sifre An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, pp. 30016–30030. Cited by: Table 1, Table 1.
  15. Intology (2026) Intology NanoGPT-Bench: evaluating autonomous research agents on the NanoGPT speedrun. Note: https://github.com/IntologyAI/NanoGPT-BenchGitHub repository Cited by: §1, §2.Source
  16. Jeddi et al. (2026) A. Jeddi, M. N. Le, H. C. Karaimer, K. G. Derpanis, and B. Taati GEAR: genetic AutoResearch for agentic code evolution. arXiv preprint arXiv:2605.13874. Cited by: §2, Table 2, Table 2, §4.
  17. Jiang et al. (2025) Z. Jiang, D. Schmidt, D. Srikanth, D. Xu, I. Kaplan, D. Jacenko, and Y. Wu AIDE: AI-driven exploration in the space of code. arXiv preprint arXiv:2502.13138. Cited by: §2.
  18. Jin et al. (2026) J. Jin, Y. Hu, K. Qiu, Q. Dai, C. Luo, G. Dong, X. Li, T. Zhao, X. Ma, G. Zhang, Z. Wu, B. Liu, Z. Yang, L. Li, L. Wang, H. Qian, Y. Zhu, and Z. Dou Toward generalist autonomous research via hypothesis-tree refinement. arXiv preprint arXiv:2606.11926. Cited by: §2, Table 2, Table 2, §4.
  19. Karpathy (2026) A. Karpathy Autoresearch: AI agents running research on single-GPU nanochat training automatically. Note: https://github.com/karpathy/autoresearchGitHub repository Cited by: §1, §2, §3.2, §3.2, §4.2.Source
  20. Kon et al. (2026) P. T. J. Kon, Q. Ding, J. Liu, X. Zhu, J. Peng, J. Xing, Y. Huang, Y. Qiu, J. Srinivasa, M. Lee, M. Chowdhury, M. Zaharia, and A. Chen EXP-bench: can AI conduct AI research experiments?. In The Fourteenth International Conference on Learning Representations, Cited by: §2.
  21. Korthikanti et al. (2022) V. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro Reducing Activation Recomputation in Large Transformer Models. arXiv preprint arXiv:2205.05198. Cited by: §4.2.
  22. Kwon et al. (2023) W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611–626. Cited by: Table 1.
  23. Lan et al. (2019) Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arXiv preprint arXiv:1909.11942. Cited by: Table 1.
  24. Liu et al. (2026) J. Liu, S. Qiu, M. Li, B. Li, H. Ji, S. Han, X. Ye, P. Xia, Z. Dong, M. Chen, C. Zhang, L. Zhang, G. Chen, H. Tu, X. Yang, L. Feng, X. Zhao, H. Chen, J. Zhou, X. Wang, W. Zhang, H. Zhu, Y. Li, J. Mei, H. Fei, J. Zhang, L. Li, L. Zhang, Y. Zhou, S. Wang, C. Xiong, J. Zou, Z. Zheng, C. Xie, M. Ding, and H. Yao AutoResearchClaw: self-reinforcing autonomous research with human-AI collaboration. arXiv preprint arXiv:2605.20025. Cited by: §1.
  25. Lu et al. (2026) C. Lu, C. Lu, R. T. Lange, Y. Yamada, S. Hu, J. Foerster, D. Ha, and J. Clune Towards end-to-end automation of AI research. Nature 651, pp. 914–919. Cited by: §2.
  26. Mattson et al. (2020) P. Mattson, C. Cheng, G. Diamos, C. Coleman, P. Micikevicius, D. Patterson, H. Tang, G. Wei, P. Bailis, V. Bittorf, D. Brooks, D. Chen, D. Dutta, U. Gupta, K. Hazelwood, A. Hock, X. Huang, D. Kang, D. Kanter, N. Kumar, J. Liao, D. Narayanan, T. Oguntebi, G. Pekhimenko, L. Pentecost, V. Janapa Reddi, T. Robie, T. St John, C. Wu, L. Xu, C. Young, and M. Zaharia MLPerf Training Benchmark. In Proceedings of Machine Learning and Systems, I. Dhillon, D. Papailiopoulos, and V. Sze (Eds.), Vol. 2, pp. 336–349. Cited by: Table 1.
  27. Nathani et al. (2025) D. Nathani, L. Madaan, N. Roberts, N. Bashlykov, A. Menon, V. Moens, M. Plekhanov, A. Budhiraja, D. Magka, V. Vorotilov, G. Chaurasia, D. Hupkes, R. S. Cabral, T. Shavrina, J. N. Foerster, Y. Bachrach, W. Y. Wang, and R. Raileanu MLGym: a new framework and benchmark for advancing AI research agents. In Second Conference on Language Modeling, Cited by: §1, §2.
  28. OpenAI (2026) OpenAI GPT-6 astra: a new generation of intelligence. Note: https://openai.com/index/gpt-6-astra/OpenAI model release announcement Cited by: §4.Source
  29. Ouyang et al. (2025) A. Ouyang, S. Guo, S. Arora, A. L. Zhang, W. Hu, C. Ré, and A. Mirhoseini KernelBench: can LLMs write efficient GPU kernels?. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, pp. 47356–47415. Cited by: §2.
  30. Press et al. (2025) O. Press, B. Amos, H. Zhao, Y. Wu, S. Ainsworth, D. Krupke, P. Kidger, T. Sajed, B. Stellato, J. Park, N. Bosch, E. Meril, A. Steppi, A. Zharmagambetov, F. Zhang, D. Pérez-Piñeiro, A. Mercurio, N. Zhan, T. Abramovich, K. Lieret, H. Zhang, S. Huang, M. Bethge, and O. Press AlgoTune: can language models speed up general-purpose numerical programs?. In Advances in Neural Information Processing Systems, Vol. 38. Cited by: §2.
  31. Shazeer (2019) N. Shazeer Fast transformer decoding: one write-head is all you need. arXiv preprint arXiv:1911.02150. Cited by: Table 1, §4.2.
  32. Shetty et al. (2025) M. Shetty, N. Jain, J. Liu, V. Kethanaboyina, K. Sen, and I. Stoica GSO: challenging software optimization tasks for evaluating SWE-Agents. In Advances in Neural Information Processing Systems, Vol. 38. Cited by: §2.
  33. Starace et al. (2025) G. Starace, O. Jaffe, D. Sherburn, J. Aung, J. S. Chan, L. Maksin, R. Dias, E. Mays, B. Kinsella, W. Thompson, J. Heidecke, A. Glaese, and T. Patwardhan PaperBench: evaluating AI’s ability to replicate AI research. In Proceedings of the 42nd International Conference on Machine Learning, pp. 56843–56873. Cited by: §2.
  34. Tang et al. (2025) J. Tang, L. Xia, Z. Li, and C. Huang AI-researcher: autonomous scientific innovation. In Advances in Neural Information Processing Systems, Cited by: §1, §2.
  35. Toledo et al. (2025) E. Toledo, K. Hambardzumyan, M. Josifoski, R. Hazra, N. Baldwin, A. Audran-Reiss, M. Kuchnik, D. Magka, M. Jiang, A. Lupidi, A. Lupu, R. Raileanu, T. Shavrina, K. Niu, J. Gagnon-Audet, M. Shvartsman, S. Sodhani, A. Miller, A. Charnalia, D. Dunfield, C. Wu, P. L. E. Saito Stenetorp, N. Cancedda, J. Foerster, and Y. Bachrach AI research agents for machine learning: search, exploration, and generalization in MLE-bench. In Advances in Neural Information Processing Systems, Vol. 38, pp. 35309–35348. Cited by: §2.
  36. Wang et al. (2026) S. Wang, B. Li, G. Zhang, W. Huang, S. Yan, and J. Li On the Residual Scaling of Looped Transformers: Stability and Transferability. arXiv preprint arXiv:2606.18524. Cited by: §1, §4.2, Table 3.
  37. Wijk et al. (2025) H. Wijk, T. R. Lin, J. Becker, S. Jawhar, N. Parikh, T. Broadley, L. Chan, M. Chen, J. M. Clymer, J. Dhyani, E. Ericheva, K. Garcia, B. Goodrich, N. Jurkovic, M. Kinniment, A. Lajko, S. Nix, L. J. Koba Sato, W. Saunders, M. Taran, B. West, and E. Barnes RE-Bench: evaluating frontier AI R&D capabilities of language model agents against human experts. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, pp. 66772–66832. Cited by: §1, §2.
  38. Zhang et al. (2025a) Y. Zhang, M. Khalifa, S. Bhushan, G. Murphy, L. Logeswaran, J. Kim, M. Lee, H. Lee, and L. Wang MLRC-Bench: can language agents solve machine learning research challenges?. In Advances in Neural Information Processing Systems, Vol. 38. Cited by: §2.
  39. Zhang et al. (2025b) Y. Zhang, C. Chen, Z. Li, T. Ding, C. Wu, D. P. Kingma, Y. Ye, Z. Luo, and R. Sun Adam-mini: Use Fewer Learning Rates To Gain More. In International Conference on Learning Representations, Cited by: §4.2.
  40. Zhao et al. (2025) B. Zhao, D. Magka, M. Jiang, X. Li, R. Raileanu, T. Shavrina, J. Gagnon-Audet, K. Niu, S. Sodhani, M. Shvartsman, A. Lupu, A. Lupidi, K. Hambardzumyan, M. Josifoski, E. Toledo, T. Foster, L. Cipolina Kun, D. Dunfield, A. Charnalia, A. Miller, O. Mac Aodha, J. Foerster, and Y. Bachrach The automated LLM speedrunning benchmark: reproducing NanoGPT improvements. In Advances in Neural Information Processing Systems, Vol. 38. Cited by: §2.
  41. Zhong et al. (2024) Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang DistServe: disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), Santa Clara, CA, pp. 193–210. Cited by: Table 1.

The following specifications use the objective names in Table 1. Displayed resource limits are rounded; each data-pool constraint fixes the available pool rather than the number of tokens consumed.

Total parameters

Why it matters. Fewer parameters shrink checkpoints and reduce model storage and transfer costs.

How we measure it. The model’s total parameter count after the fixed 600-second training schedule.

Paper table
OptimizeMinimize total parameters.
Maintain qualityValidation BPB << 1.05 (fixed quality threshold)
Stay within limitsTraining FLOPs ≤\leq239.1 M/token Peak training memory ≤\leq44.0 GiB Available training-data pool Fixed 631.2 M-token pool
View original tableOriginal table

Training tokens to quality

Why it matters. Consuming fewer tokens reduces the data processing needed to train a model of the required quality.

How we measure it. Total training tokens consumed by the first scheduled validation check with BPB below 1.05, including warmup updates, gradient accumulation, and repeated use of data. This is token consumption, not the number of unique dataset positions visited.

Paper table
OptimizeMinimize consumed training tokens.
Maintain qualityValidation BPB << 1.05 (fixed quality threshold)
Stay within limitsTraining FLOPs ≤\leq239.1 M/token Total parameters ≤\leq50.3 M Peak training memory ≤\leq44.0 GiB Available training-data pool Fixed 631.2 M-token pool
View original tableOriginal table

Training FLOPs

Why it matters. Fewer FLOPs reduce training computation independently of hardware and implementation speed.

How we measure it. Forward-and-backward FLOPs per token, measured on the uncompiled model using executed operations and registered counting formulas. The count includes matrix, convolution, and attention work, but excludes optimizer updates and unregistered operations such as elementwise operations and normalization.

Paper table
OptimizeMinimize FLOPs per token.
Maintain qualityValidation BPB << 1.05 (fixed quality threshold)
Stay within limitsTotal parameters ≤\leq50.3 M Peak training memory ≤\leq44.0 GiB Available training-data pool Fixed 631.2 M-token pool
View original tableOriginal table

Peak training memory

Why it matters. Lower memory lets training fit on devices with less capacity or use larger batches.

How we measure it. Peak PyTorch-allocated GPU memory from process setup through the reporting read, including FLOPs probing, training, and validation under the fixed 600-second schedule. Validation uses a fixed batch of 128 sequences of 2,048 tokens and can determine the peak. This is allocated memory, not allocator-reserved memory or total device usage.

Paper table
OptimizeMinimize peak allocated memory.
Maintain qualityValidation BPB << 1.05 (fixed quality threshold)
Stay within limitsTraining FLOPs ≤\leq239.1 M/token Total parameters ≤\leq50.3 M Available training-data pool Fixed 631.2 M-token pool
View original tableOriginal table

Training time

Why it matters. Shorter training time reaches the required quality sooner and captures hardware and implementation effects that FLOPs miss.

How we measure it. Measured training time at the first scheduled validation check with BPB below 1.05. Startup, compilation, evaluation, and the first ten warmup updates are excluded; each subsequent update includes all accumulation microbatches and the optimizer step, with GPU synchronization at its boundaries.

Paper table
OptimizeMinimize measured training time.
Maintain qualityValidation BPB << 1.05 (fixed quality threshold)
Stay within limitsTraining FLOPs ≤\leq239.1 M/token Total parameters ≤\leq50.3 M Peak training memory ≤\leq44.0 GiB Available training-data pool Fixed 631.2 M-token pool
View original tableOriginal table

Training updates

Why it matters. Fewer updates indicate faster optimization progress, separate from the duration or arithmetic cost of each update.

How we measure it. Total logical optimizer updates completed by the first scheduled validation check with BPB below 1.05, including warmup updates. Accumulation microbatches within one update are not counted as separate updates.

Paper table
OptimizeMinimize optimizer updates.
Maintain qualityValidation BPB << 1.05 (fixed quality threshold)
Stay within limitsTraining FLOPs ≤\leq239.1 M/token Total parameters ≤\leq50.3 M Peak training memory ≤\leq44.0 GiB Available training-data pool Fixed 631.2 M-token pool
View original tableOriginal table

Live-state memory (KV cache)

Why it matters. Retained state grows with the live context and limits either context length or concurrent requests during generation.

How we measure it. The increase in PyTorch-allocated GPU memory while retaining one request state after a 1,536-token prefill and 512 decoding steps, with capacity 2,048. Measurement follows a discarded warmup request and constructs the state with CUDA graph capture disabled. It measures retained allocation rather than transient peak memory or a logical tensor-size sum, and includes allocator rounding (approximately 1 MiB in the reference measurement).

Paper table
OptimizeMinimize long-prompt retained GPU allocation.
Maintain qualityValidation BPB << 1.05 (fixed quality threshold)
Stay within limitsRequest latency ≤\leq750 ms Training FLOPs ≤\leq239.1 M/token Total parameters ≤\leq50.3 M Peak training memory ≤\leq44.0 GiB Available training-data pool Fixed 631.2 M-token pool
View original tableOriginal table

Decode latency

Why it matters. Lower decode latency produces generated tokens faster after the prompt has been processed.

How we measure it. Median latency for a one-token prefill followed by 512 decoding steps, measured over 30 timed passes after three warmups. The timer includes prefill and synchronized GPU completion, but excludes state allocation and reset.

Paper table
OptimizeMinimize latency.
Maintain qualityValidation BPB << 1.05 (fixed quality threshold)
Stay within limitsRetained GPU state allocation at 513 tokens ≤\leq10 MiB Training FLOPs ≤\leq239.1 M/token Total parameters ≤\leq50.3 M Peak training memory ≤\leq44.0 GiB Available training-data pool Fixed 631.2 M-token pool
View original tableOriginal table

Long-prompt latency (request latency)

Why it matters. Lower request latency reduces the full delay of processing a long context and generating a response.

How we measure it. Median latency for a 1,536-token prefill followed by 512 decoding steps, measured over 30 timed passes after three warmups. The timer includes prefill and synchronized GPU completion, but excludes state allocation and reset.

Paper table
OptimizeMinimize latency.
Maintain qualityValidation BPB << 1.05 (fixed quality threshold)
Stay within limitsRetained GPU state allocation at 2,048 tokens ≤\leq40 MiB Training FLOPs ≤\leq239.1 M/token Total parameters ≤\leq50.3 M Peak training memory ≤\leq44.0 GiB Available training-data pool Fixed 631.2 M-token pool
View original tableOriginal table

Shared inference correctness requirements. The three inference tasks supply ground-truth tokens at each decoding step and compare the resulting next-token distributions with the same model’s full forward pass. For both the one-token and 1,536-token prefill settings, the maximum total-variation distance across all 513 scored positions must not exceed 0.05. The returned request-state object must own all retained KV data, including compressed representations or replacement token history, and these data must remain on the GPU throughout the request. Memory constraints use the same retained-allocation procedure as the KV-cache objective, with capacities of 513 and 2,048 for the short- and long-prompt settings, respectively.

This section specifies the integration interface and the two reference search policies used in Section 4. Arena provides execution, measurement, and scoring; the baselines determine proposals and retention. Their selection rules are harness choices, not requirements imposed on other integrations.

Minimal interface. A harness must inspect the provided workspace and task definition, modify only the permitted program, submit experiments through Arena, read standardized results, and choose subsequent candidates. Arena defines the research problem and evaluation environment while leaving the research strategy open: an integration may retain an incumbent, beam, population, hypothesis tree, or another search state.

Recorded artifacts. Each run records the reference program, proposed ideas, candidate implementations, execution outcomes, measured metrics, admissibility decisions, and search lineage. The terminal score is computed from measured admissible outcomes; the accompanying record supports audits and post-hoc trace analysis. Charged-experiment accounting follows Section B.4, independently of harness-side bookkeeping.

Arena prepares the environment and evaluates the reference program. The harness receives its source and measurements, the task’s objective, BPB threshold, resource constraints, measurement procedures, experiment budget, and a workspace for candidate programs and research records.

Sequential Search starts from one incumbent and proposes one candidate per round. Beam Search retains up to four programs and proposes two modifications per retained program in each generation. A full generation therefore contains eight candidate experiments; selection occurs after all outcomes are available.

A persistent controller coordinates an ideator, executor, and reviewer. The implementation calls the reviewer an evaluator, distinct from Arena’s evaluator. Agent sessions are fresh for each round or generation. Instructions, source code, task rules, and saved research history are supplied to each session, so continuation does not rely on conversational memory.

The ideator inspects the current code and prior experimental history, then records a proposal, rationale, timestamp, and position in that history. Edits may change architecture, attention, optimization, data processing, or training within the permitted boundary. A proposal can coordinate several edits implementing one mechanism; Sequential Search also requests follow-up experiments that could separate their contributions.

The executor implements each proposal in an isolated copy of its parent program and checks the edits before Arena’s benchmark evaluation.

Arena checks the submitted source, protected files, available launch budget, and hardware, then runs training and task-specific evaluation. It records source, measurements, execution status, and logs. Successful execution alone does not establish admissibility.

Sequential Search uses one worker; Beam Search uses four. Parallel submission does not alter accounting: each candidate is one experiment. A full eight-candidate generation needs eight available slots and consumes eight launches if all reach chargeable GPU work, although at most four run simultaneously. Local CPU checks do not replace candidate training through Arena.

The budget is 100 charged launches, including the reference evaluation. A recorded GPU-work-start marker determines charging. Once GPU work begins, crashes, out-of-memory errors, and task failures still consume budget. A resolved launch without the marker is not charged; an unresolved launch reserves a slot. A normally charged reference therefore leaves at most 99 candidate experiments. The harness reads Arena’s remaining budget rather than counting its own rounds or generations. Failed experiments remain recorded.

The reviewer compares the proposal, parent and candidate code, measurements, and logs, then records feedback. Both baselines exclude failed or constraint-violating candidates from further development. Other harnesses may continue refining them, but only admissible results contribute to Arena’s score.

Unchanged-target attempts. An efficiency improvement may increase BPB while remaining below the quality threshold. Both baselines also permit limited attempts to improve quality without changing the target. Target equality means exact equality of recorded values, regardless of the proposal’s description. Such attempts consume the allowance even when rejected.

Sequential Search. An admissible candidate replaces the incumbent when it strictly improves the target. At exactly equal target value, it may replace the incumbent while the allowance remains if it improves either the task-defined tie-break metric or BPB. Otherwise the incumbent is retained. The allowance starts at four unchanged-target attempts and resets only after an accepted strict target improvement; accepting a quality improvement alone does not reset it.

Beam Search. Parents remain fixed throughout a generation. After every outcome is resolved, the reviewer pools parents and new candidates, removes exact program duplicates, and excludes failures, constraint violations, and missing required ranking measurements. Parents need not be replaced when their modifications fail.

Up to two unchanged-target attempts are permitted per generation only if the best retained target strictly improved during the preceding two completed generations. This allowance is determined before proposal generation. When permitted, a parent and modification with equal target are reduced to the admissible member with lower BPB. The remaining programs are ranked by target, ordered task tie-break metrics, and finally BPB, retaining up to four. Remaining ties favor lower complexity, lower resource use, and then an older program where evidence is available. A read-only helper computes identities, admissibility checks, and ranking values; the reviewer applies the selection policy.

Saved state retains programs, measurements, decisions, feedback, and the remaining unchanged-target allowance, together with Beam generation membership and outcomes. Rejected and failed experiments remain in the history. Continuation reconciles outstanding requests with Arena and completes pending review before new proposals; recovery itself is not a new experiment.

Sequential Search normally stops when the budget is exhausted. Beam Search also stops when the remaining budget cannot cover two candidates per retained program; it does not shrink the generation or select from a partially evaluated one. Both record the stopping reason and unused budget.

Arena independently ranks all admissible recorded results and reports the best program, which need not be the final harness-selected endpoint. Computing this result requires no additional training.

This catalogue presents nine representative findings, detailing their mechanisms, objectives, and relationships to prior work.

Table C.1 gives mechanisms, objectives, and comparisons with prior work. These examples use the BPB thresholds specified in Table 7. Correspondences identify related research questions and technical differences, not necessarily exact rediscoveries. Endpoints combine multiple changes and do not isolate individual mechanisms’ contributions.

Table 7: Nine findings from Auto-Research Arena: six research findings (a–f) and three infrastructure engineering findings (g–i). The latter address low-precision updates, activation memory, and decode execution in large-model infrastructure. Blue highlights searched changes.