HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry

A harness configuration is a tuple $\mathcal{H}=(c_{1},c_{2},\ldots,c_{9})$, where each $c_{i}\in\mathcal{C}_{i}$ instantiates one of the nine behavioral dimensions (Section 3.3): model selection ($c_{1}$), context assembly ($c_{2}$), memory management ($c_{3}$), tool ecosystem ($c_{4}$), execution environment ($c_{5}$), evaluation and reward ($c_{6}$), control and safety ($c_{7}$), observability ($c_{8}$), and training bridge ($c_{9}$). Each $\mathcal{C}_{i}$ is the set of valid processor configurations for dimension $i$, constrained by hook-type contracts and singleton-group exclusion (Section 3.2).

A harness edit is a function $e:\mathcal{H}\to\mathcal{H}$ that modifies one or more dimensions while preserving type contracts. The action space $\mathcal{E}$ is discrete but open-ended: each edit is a code-level artifact (new processor source, modified prompt template, reconfigured tool registry, or control-flow rewrite) generated by the meta-agent LLM, not selected from a pre-enumerated set. Combinatorial explosion is managed not by exhaustive search but by the LLM’s generative capacity—the Planner proposes edits from trace-grounded hypotheses—and by type constraints that prune invalid compositions at generation time.

The operational mirror is the tuple $(\mathcal{H},\mathcal{E},\mathcal{R},\mathcal{T})$, where $\mathcal{H}$ is the harness-configuration space (states), $\mathcal{E}$ is the code-level edit space (actions), $\mathcal{R}:\mathcal{H}\times\mathcal{E}\to\mathbb{R}$ maps a configuration–edit pair to a scalar reward (verifier scores aggregated over an adaptation batch), and $\mathcal{T}$ is the trace store that provides structured feedback beyond the scalar signal. This tuple forms an MDP at the harness level: harness configurations are states, typed edits are actions, execution traces plus verifier scores constitute feedback, and a deterministic acceptance gate governs state transitions.

Let $\mathcal{H}_{t}$ denote the harness configuration at iteration $t$ (the model $\mathcal{M}$ is fixed throughout evolution), and let $\mathcal{T}_{t}$ denote the trace store accumulated from all previous executions. We define the symbolic state as $s_{t}=(\mathcal{H}_{t},\mathcal{T}_{t})$. A harness-update policy $\pi_{\mathrm{evo}}$ selects an action $a_{t}\sim\pi_{\mathrm{evo}}(\cdot\mid s_{t})$, where $a_{t}\in\mathcal{E}$ is a code-level edit drawn from the builder algebra. Applying this edit yields a candidate harness $\widetilde{\mathcal{H}}_{t}=a_{t}(\mathcal{H}_{t})$. Running the candidate on an adaptation batch (with the fixed model $\mathcal{M}$) produces new traces $\Delta\mathcal{T}_{t}$ and per-task verifier scores $r_{t}$. A deterministic acceptance operator $U(\widetilde{\mathcal{H}}_{t},\mathcal{T}_{t},r_{t})$ then either commits the candidate ($\mathcal{H}_{t+1}=\widetilde{\mathcal{H}}_{t}$) or rejects it ($\mathcal{H}_{t+1}=\mathcal{H}_{t}$), enforcing the seesaw constraint: the candidate must not regress any previously solved task recorded in $\mathcal{T}_{t}$. In both cases, the trace store grows: $\mathcal{T}_{t+1}=\mathcal{T}_{t}\cup\Delta\mathcal{T}_{t}$.

This MDP operates at the harness level: within a single task, $\mathcal{H}_{t}$ (together with the fixed $\mathcal{M}$) determines the agent’s behavior; across iterations, the harness-update policy $\pi_{\mathrm{evo}}$ modifies the harness. AEGIS realizes $\pi_{\mathrm{evo}}$ as a four-stage pipeline (Digester, Planner, Evolver, Critic) that maps $s_{t}$ to candidate edits through trace compression, adaptation planning, edit generation, and candidate assessment.

In standard RL, reward hacking [10] exploits loopholes in the reward signal without genuine task completion. Symbolic harness evolution amplifies this risk because the evolver can target the verification protocol directly: embedding benchmark answers into prompts, exploiting format regularities in the verifier, or introducing a processor that rewrites outputs to match verifier expectations.

Catastrophic forgetting [15] occurs when improving performance on one region of the task distribution harms another. In symbolic harness evolution, an edit that repairs failure pattern $A$ can silently regress pattern $B$, because effects propagate through shared context, tools, memory policies, and control rules. Without explicit regression checking, an evolver conditioned only on failing-task traces cannot distinguish local gain from global regression.

Under-exploration [16] manifests as a bias toward low-risk local edits: prompt rephrasing, tool-description tuning, or minor control-flow tweaks. These edits are cheap to generate and frequently pass gating without regressing solved tasks, biasing subsequent Planner hypotheses toward the same edit neighborhood. Structural changes (decomposing one agent into several, replacing the control strategy, or adopting a new memory architecture) require deliberate hypothesis formation and rarely emerge from trace-conditional local repair. Without a mechanism to propose edits beyond the immediate failure neighborhood, the system plateaus once local edits are exhausted.

Symbolic harness evolution inherits the structural risks of RL (reward hacking, catastrophic forgetting, and under-exploration), and AEGIS addresses each with a dedicated mechanism: the Critic (reward hacking), the deterministic gating layer (catastrophic forgetting), and the Planner (under-exploration).

A single iteration on GAIA (103 tasks, pass@2) generates ${\sim}$10M tokens of raw traces: model reasoning steps, tool invocations with their outputs, and timing metadata. Passing this volume directly to downstream stages exceeds context limits, yet naive truncation discards diagnostic signal. The Digester compresses each task’s traces into a structured per-task summary: binary outcome, failure category (if any), implicated component identifiers, and supporting evidence excerpts. It also provides cross-iteration continuity: each task’s summary links to its history of prior outcomes and shipped edits, enabling the Planner to distinguish persistent failures from transient noise.

The Planner receives the Digester’s output (task-level summaries enriched with cross-iteration history) and constructs an adaptation landscape: which tasks are failing, what edits have been attempted, which components are implicated, and which edit types (prompt, tool, processor, configuration) remain untried. This stage is the primary defense against under-exploration: by constructing the landscape before edit generation, it prevents the pipeline from converging on trace-conditional local repair, ensuring that structural changes (tool additions, processor rewrites, memory-policy redesigns) are considered alongside incremental prompt edits.

Given the Planner’s adaptation landscape, the Evolver produces one or more candidate harnesses $\{\widetilde{\mathcal{H}}_{t}^{\,k}\}_{k=1}^{K_{t}}$, each specified as a typed builder operation on the current harness $\mathcal{H}_{t}$. Each candidate carries a change manifest: the edited components, the intended behavioral effect, and the tasks expected to improve or regress. When introducing new processor code, the Evolver must also provide a smoke test confirming that the processor instantiates and runs on synthetic input without raising exceptions. The builder algebra guarantees type-safety (every candidate satisfies hook-type contracts and processor-composition rules) but not behavioral safety; an edit that type-checks may still produce non-local behavioral effects, detectable only by the Critic and gating layer.

The Critic defends against reward hacking; the deterministic gating layer defends against catastrophic forgetting. The Critic evaluates each candidate by comparing its change manifest against trace evidence and assessing whether edits risk non-local effects through shared state or control flow. When gaps are detected, it issues a single revision request to the Evolver. After at most one revision cycle, the Critic returns either no_op or an ordered ship_ranking. The deterministic gate then applies acceptance checks in sequence: manifest completeness, configuration normalization (ensuring the candidate is in canonical form), build or smoke tests (when applicable), and the seesaw constraint (regression check on previously passing tasks; Section 4.1). The first failing check halts the sequence; passing candidates are committed and failing ones archived with their rejection reason. This decouples LLM judgment from acceptance: regardless of the Critic’s recommendation, only deterministic checks govern shipping.

Language-model subagents explore, hypothesize, and propose; typed structure and deterministic gates determine what ships. This separation ensures that safety properties (no regression, no unaudited edits) hold regardless of LLM subagent failure modes.

Harness evolution proceeds as in Section 4, drawing on the replay buffer $\mathcal{B}$ for trace evidence. The principal difference from standalone AEGIS is that $\mathcal{B}$ contains trajectories from multiple model checkpoints $\mathcal{M}_{0},\mathcal{M}_{1},\ldots,\mathcal{M}_{t}$, exposing the Digester to behavioral variation from both model updates and harness edits.

The key design choice is the cross-harness grouping criterion (formalized in Section 5.3): all trajectories sharing a task identifier form one GRPO group regardless of which harness or model checkpoint produced them, so that within-group variation reflects strategy differences rather than sampling noise alone.

The harness update makes discrete structural changes (adding a tool, replacing a control processor, restructuring the prompt) that cannot be expressed as parameter updates. The model update makes fine-grained behavioral adjustments (when to invoke which tool, how to phrase a query, when to terminate) that depend on high-dimensional in-context state and cannot be captured by symbolic specification. The harness defines coarse-grained strategy architecture; the model learns to exploit it.

Cross-harness GRPO performs task-level alignment: trajectories from different harness versions are grouped by task identity and compared by verifier reward alone. No action-level alignment is required, so harness versions with incompatible action spaces (different tool schemas, different prompt structures, different control-flow processors) coexist in the same group without conflict. When computing the policy gradient, each trajectory $\tau_{i}$ is replayed under the harness version $\mathcal{H}_{k}$ that produced it: the model’s log-probabilities $\pi_{\theta}(\tau_{i}\mid x)$ are evaluated against the prompt, tool schema, and observation context that $\mathcal{H}_{k}$ would have constructed at each turn. The GRPO gradient thus operates entirely on model output tokens conditioned on harness-specific context, rather than on harness structural actions or environment transitions. This design decouples harness evolution (which may freely alter the action space across versions) from model training (which only requires token-level log-probabilities under each trajectory’s own harness context).

The group-relative advantage is:

where $r_{i}$ is the reward for trajectory $\tau_{i}$, and $\mu(\mathcal{G}_{x})$, $\sigma(\mathcal{G}_{x})$ are the within-group reward mean and standard deviation. The evolving harness acts as a structured exploration operator for the model’s RL: each new version injects a distinct mode of behavior into the task’s sampling distribution, and the advantage in Eq. 3 commits the model toward whichever modes the verifier scores highest. The exploration breadth that single-policy sampling cannot provide is thus supplied by the evolving scaffold itself.

The policy objective to maximize is:

where

is the importance-sampling ratio between the current policy $\mathcal{M}_{k}$ and the checkpoint $\mathcal{M}_{d}$ that generated $\tau_{i}$ (Eq. 1), $\epsilon_{c}$ is the clipping threshold, and $\beta\,D_{\text{KL}}(\pi_{\theta}\|\pi_{\text{ref}})$ penalizes divergence from the fixed reference model $\pi_{\text{ref}}$. The behavior policy $\pi_{\theta_{\text{old}}}$ in the ratio and the reference policy $\pi_{\text{ref}}$ in the KL term are distinct: $\pi_{\text{ref}}=\mathcal{M}_{0}$ is fixed throughout training, while $\pi_{\theta_{\text{old}}}$ varies per trajectory and must be recovered from the buffer (Section 5.4).

The importance ratio (Eq. 5) corrects the gap between $\pi_{\theta}$ and the checkpoint $\mathcal{M}_{k}$ that produced $\tau_{i}$. Since $\mathcal{M}_{k}$ varies across the buffer, $\pi_{\theta_{\text{old}}}(\tau_{i})$ cannot be recovered from any single model: we materialize it at buffer insertion via one forward pass under $\mathcal{M}_{k}$, cache the token-level log-probabilities on disk, and reuse them at every gradient step. This decouples the cached behavior log-probabilities from the current log-probabilities $\pi_{\theta}(\tau_{i})$ recomputed each step.

FIFO eviction caps the buffer at $C$ trajectories; with $s$ samples per round the maximum model-version lag is $\lfloor C/s\rfloor$ rounds, so every cached $\pi_{\theta_{\text{old}}}$ originates within a bounded window of $\pi_{\theta}$ and the policy that generated a trajectory never differs greatly from the one being updated. The same window bounds harness staleness, so the cross-harness groups (Eq. 2) mix only recent scaffold versions, and the model is never trained predominantly against an obsolete harness.

The dominant cost of agentic RL is the rollout (executing the agent in the environment: model decoding, tool calls, and verification), not the gradient update. In co-evolution a single round of exploration produces one set of trajectories that serves both updates: the same traces drive the AEGIS harness update (Section 4) and, through the shared buffer (Section 5.1), the cross-harness GRPO model update. GRPO consumes these trajectories by replay and issues no rollouts of its own. The marginal cost of adding the model update is therefore confined to (i) one cached forward pass per trajectory to record $\pi_{\theta_{\text{old}}}$ and (ii) the gradient steps themselves, both of which are rollout-free. No trajectory is generated solely to train the model. Joint optimization is therefore economical: it buys model improvement for the price of offline training compute alone, without any rollouts beyond those harness evolution already performs.

As summarized in Table 3, we evaluate on five benchmarks spanning multi-step retrieval, embodied planning, web interaction, multi-turn dialogue, and software engineering. Unless otherwise noted, each experiment runs for up to $T{=}15$ evolution rounds with early stopping after $P{=}3$ consecutive rounds without a shipped edit. The full task set is evaluated every round (no subsampling). The meta-agent token budget varies by benchmark (100M–175M total) but is held constant across task agents within a benchmark.

We distinguish two roles: the meta-agent (Claude Opus 4.6 unless otherwise noted) drives the AEGIS evolution loop; the task agent runs under the evolved harness to solve benchmark tasks. Task agents span three families (Claude Sonnet 4.6, GPT-5.4, and Qwen3.5-9B) to test whether a single meta-agent can evolve effective harnesses across model families.

(1) Static Harness: a HarnessX configuration constructed from published benchmark-specific prompts and tool definitions, held fixed across all rounds. (2) Claude Code SDK (CC SDK)$^{1}$: a single-agent evolver (one LLM session per round) that replaces the four-stage pipeline while retaining the same infrastructure and round budget, isolating AEGIS’s multi-stage architecture from the shared infrastructure (Section 6.4). This baseline also serves as a proxy for monolithic evolvers such as SICA [31].

Task success rate (%) under the benchmark-specific verifier. Each task receives two independent attempts per round (pass@2: solved if either succeeds), reducing sampling noise while preserving a binary per-task signal for the seesaw constraint (at the cost of masking sub-threshold success-probability drift; Section 6.3).

All reported gains are measured on the same task set used for evolution; held-out generalization to unseen tasks is not evaluated in this work.

The averaged $\tau^{3}$-Bench gains mask substantial per-domain variation. GPT-5.4 gains +25.4% on Telecom (67.5% $\to$ 93.0% at R2) and +9.7% on Retail (84.2% $\to$ 93.9% at R6). However, Sonnet 4.6 on Telecom regresses $-$14.0% in a single round (R7) due to accumulated same-type edits, recovering by R9 (Section 6.6). This illustrates a structural limitation of per-edit gating: sub-threshold coupling from consecutive same-type edits accumulates undetected until a tipping point triggers visible regression.

On SWE-bench Verified (GPT-5.4), evolution peaks at 63.6% (R3, +18.2%) but degrades to 50.9% by R5 ($-$12.7% from peak); final accuracy still exceeds the static baseline by +5.4%. Two factors accelerate degradation on this benchmark: (1) with only 55 tasks, each task flip shifts aggregate accuracy by ${\sim}$1.8% (vs. ${\sim}$1.0% at $n{=}103$), so fewer regressions suffice to produce visible decline; and (2) structural code edits have a broader blast radius than prompt edits. This parallels the GAIA GPT-5.4 stagnation: both cases motivate the variant-isolation strategy evaluated in Section 6.3.

At R10, the pipeline shipped a composite edit (tool + prompt + configuration) whose manifest predicted improved retrieval. The edit passed the seesaw constraint and raised accuracy from 74.8% to 79.6%. Trace analysis at R11 revealed that the tool genuinely fixed retrieval for most newly passing tasks, but a subset passed by exploiting format regularities in the verifier rather than performing actual retrieval. The Planner flagged this pathway at R12, and the resulting edit introduced a guard restricting the tool to tasks whose output could be cross-checked against a second retrieval path.

Evolution on Telecom shipped same-type prompt/processor edits across five consecutive rounds (R2–R6), each appending a “reminder” rule. Compliance rose from 89.5% to 100% at R4, then regressed to 94.7% by R6 as later rules conflicted with earlier ones. The R7 Critic flagged the concentration risk (“All 5 prior ships occupy the same bucket: [prompt, processor]”) but still approved the edit for shipping because ship-prediction accuracy remained high (R2–R6: 23/24, 5/6, 4/5, 7/7, 2/3) and no regressions were recorded. The sixth reminder degraded compliance from 94.7% to 80.7% ($-$14.0%) via cross-rule conflicts that destabilized previously passing tasks. This regression evaded the seesaw constraint because pass@2 registers only per-task binary flips, not sub-threshold coupling. The pipeline self-corrected by R9 once the Planner diagnosed the concentration pattern and proposed a structural edit that replaced the conflicting reminder stack.

Between R4 and R7, the pipeline shipped predominantly prompt-level edits, yielding $<$1% gain per round. Ship-prediction accuracy (the fraction of manifest-predicted task flips that materialize) dropped from 80% (R3) to 0% (R7), signaling prompt-space exhaustion. The sole structural edit in this window (a processor-level change at R6) achieved only 14% ship-prediction accuracy (1/7 predicted flips materialized), suggesting that the Planner lacked sufficient structural-edit history to calibrate hypotheses beyond the prompt neighborhood.

All three pathologies predicted by the operational mirror appear in practice. The pipeline detected and mitigated reward hacking within two rounds (R10–R12). Decaying ship-prediction accuracy diagnosed under-exploration (R4–R7). The catastrophic-forgetting case exposes a structural limitation of per-edit gating: sub-threshold coupling accumulates undetected until it exceeds the per-task detection threshold (Telecom R7). On $\tau^{3}$-Bench Telecom, the pipeline self-corrected (R8–R9) because the failure was localized to one domain; on GAIA (GPT-5.4), the same mechanism produces sustained stagnation ($\Delta{=}0.0$) because conflicting edits prevent any net gain. Section 6.3 shows that variant isolation resolves this by confining edits to task-specific clusters.

The GAIA benchmark [25] poses real-world questions that are conceptually simple for humans but require an agent to compose multiple actions (web search, file extraction, multimodal interpretation, arithmetic) and evaluates via exact match against a reference answer. This benchmark stresses open-ended tool-based reasoning, where the harness dictates how evidence is collected and synthesized.

The ALFWorld benchmark [35] involves embodied instruction following where a text-based agent commands a simulated robotic agent in household settings. Given a natural-language goal (e.g., “Put a cooled apple in the microwave”), the agent navigates rooms, identifies objects, and manipulates them via textual actions; performance is measured by goal-completion rate. This benchmark stresses multi-step planning and grounded search under a tight step budget. We use the 134 tasks from the valid-unseen set, spanning six task types: pick-and-place, pick-two-and-place, look-at-in-light, and three transform-then-place variants (heat, cool, clean).

WebShop [44] is a web-interaction benchmark in which an agent acts as a customer in a simulated online store. Given a textual product description, the agent must search, browse product pages, select the best-matching item, and purchase it; scoring reflects how well the chosen product satisfies the request. We evaluate on 100 instances sampled with a fixed seed, each run as an independent shopping session.

$\tau^{3}$-Bench [45] is a multi-turn dialogue benchmark in which the agent plays a customer-service assistant that must satisfy a user request while obeying an explicit domain policy. Performance is measured by rule compliance across the full conversation. The benchmark stresses dialogue-policy adherence: the harness must prevent the agent from agreeing to disallowed actions across many turns. For evaluation, we select three domains from the benchmark: Retail, Airline, and Telecom.

SWE-bench Verified [13] is a human-validated subset of SWE-bench in which each task requires an agent to resolve a real GitHub issue by editing the corresponding repository so that the project’s hidden test suite passes. This benchmark stresses repository-level code editing: navigating a large codebase, localizing the relevant fault, implementing a patch, and avoiding regressions in existing tests. For evaluation, we sample a 55-task subset from SWE-bench Verified and measure performance by patch resolution.

For a configuration evaluated on a task set $D$ with $n$ rollouts per task, let $r_{i,j}\in\{0,1\}$ denote the binary outcome of rollout $j$ on task $i$. Let $c_{i}=\sum_{j=1}^{n}r_{i,j}$ be the number of successful rollouts for task $i$. We report pass@$k$ using the standard unbiased estimator, i.e., the probability that at least one of $k$ sampled rollouts solves the task:

All evolution curves use pass@2 as the primary metric: each task receives two independent rollouts and is solved if either succeeds. This reduces sensitivity to single-rollout stochasticity while preserving a strict task-level criterion. Rollouts terminated by infrastructure failures (sandbox crashes, API timeouts) count as failures rather than being excluded, keeping results comparable to official leaderboard protocols.

Prior to round 10, the success rate stood at $74.8\%$, having dropped from its peak of $77.7\%$ due to a regression in round 9. The Digester’s trace analysis revealed a systematic failure pattern: every Wikipedia fetch in round 10’s traces returned zero characters. WebFetch employs a browser if the website requires JavaScript support, but Wikipedia’s new frontend fails to load correctly, timing out or returning an empty body. The traces make it plain: within task db4fd70a (number of stations in a rail line), db4fd70a_r0.jsonl#step_0 and #step_1 report that Wikipedia WebFetch fetches return 0 chars; similarly, within f0f46385 (ASEAN members’ membership status), three consecutive WebFetch calls return 0 chars; ten separate attempts across the round returned empty responses.

The Digester grouped the 23 failed tasks by failure mode, surfacing a critical tool-level issue that the round-9 Critic had already flagged: the tools component had not shipped a fix in nine consecutive rounds despite source-access failures appearing since round 1. The Planner received two targets: (1) resolve the persistent tool-level source-access failures, and (2) revert the prompt and budget-processor changes responsible for the round-9 regression.

The Evolver suggested C-R10-02, covering three buckets. (i) tools: new WikiTextFetch tool, avoiding the browser altogether by employing the MediaWiki API endpoint; returns complete text of the article, 10,529 chars in case of the rail line, 80,028 for ASEAN. (ii) prompt: a single sentence in the tool usage section instructing the agent to use WikiTextFetch before looking up Wikipedia articles. (iii) config: restore the round-8 baseline configuration, register WikiTextFetch, and remove the problematic budget processor. The manifest’s capability evidence includes a Level-2 round-trip check (content serialized as a 10,529-char string by the provider); the attribution signature requires at least one WikiTextFetch call.

The Critic approved only C-R10-02, rejecting the competing revert-only candidate C-R10-01. Its rationale covered three aspects. (i) Interaction: C-R10-02 is a strict superset of C-R10-01; both restore the round-8 baseline, so the budget-processor removal is intentional, not an accidental overlap. (ii) Round-trip evidence: the Critic verified Level-2 evidence (tool output arrives as a full string, not a truncation marker) before accepting any tools-bucket candidate. (iii) Portfolio: this is the first tools-bucket ship in ten rounds, and the trace evidence of persistent zero-char returns justifies the intervention that round-9 flagged.

Post-shipping, the GAIA pass rate increased from $74.8\%$ at R9 to $79.6\%$ at R10 (+$4.9$pp, five tasks changed to pass), the greatest improvement during the entire run; five of the seven tasks the tool was predicted to affect flipped to pass (hit rate $0.71$, the highest for any ship across all 19 runs). The improvement mainly occurred at Levels 2 (+$4$ tasks) and 3 (+$2$). Since the tool triggered its target tasks, the attribution condition was satisfied.

Figure 7 summarizes the same edit as a manifest card: the raw YAML above is what the loop logs, the card is the human-facing reading of it.

Panel (a) summarizes the failure clusters accumulated across the GAIA run. The dominant cluster is blocked-source ($39\%$), where the agent cannot retrieve the required evidence because pages return empty content, require JavaScript rendering that times out, or contain incomplete information. Reasoning failures ($33\%$) follow, covering tasks that require multi-hop inference, disambiguation of similar entities, numerical computation, or precise interpretation of underspecified queries. Figure/visual failures ($11\%$) arise when the answer depends on information embedded in images, maps, or diagrams that text extraction alone cannot capture. Document/table parsing failures ($11\%$) occur when evidence is locked in PDFs, structured tables, or semi-structured formats and the agent either misparses the layout or overlooks relevant cells. Scope ambiguity ($6\%$) covers queries with multiple valid interpretations, where the agent answers a related but incorrect reading. Together, these clusters indicate that GAIA failures concentrate in evidence retrieval, multi-step reasoning, visual grounding, structured-document extraction, and query disambiguation.

Panel (b) shows GAIA is the only benchmark where all four levers see substantial use; the Sonnet run alone shipped 11 prompt, 7 processor, 7 config, and 6 tools edits, because its failure set spans tool, prompt, and config problems simultaneously. The three models nonetheless diverge in a way that tracks their starting competence: Sonnet, with the most rounds, sweeps every lever; GPT-5.4 leans hardest on prompt ($45\%$ of its ships) and barely touches config, since its reasoning is already strong enough that the remaining gains are mostly instruction-following; Qwen3.5’s short run concentrates its few ships and, strikingly, lands its single tools ship at the highest yield of any cell. The shared logic is prompt-first for the behavioral clusters, with the scarce tools lever reserved for the one mechanical cluster prose cannot touch.

Panel (c) shows which failure clusters were reduced by evolution. The largest improvement comes from the blocked-source cluster: the tool edit that introduced WikiTextFetch replaced unreliable browser-based Wikipedia fetching with a MediaWiki API call, reducing failures caused by empty or incomplete page retrieval. Prompt edits mainly targeted the reasoning cluster by encouraging more explicit verification, which contributed to steady gains across rounds. By contrast, figure / visual and document / table parsing failures remained harder to reduce, because they require information extraction from images, figures, PDFs, or structured tables. Overall, GAIA improves through a combination of tool edits that fix retrieval failures and prompt edits that reduce reasoning errors, while residual errors concentrate in visual and document-heavy tasks.

Panel (a) summarizes the main failure clusters observed on ALFWorld. The dominant cluster is search / step-ceiling ($89\%$), which covers episodes where the agent either searches rooms or receptacles in an inefficient order, or reaches the step limit before completing long interaction chains such as deep search or transform-then-place tasks. Prompt-rule side-effect failures ($7\%$) occur when an added heuristic improves some tasks but unintentionally restricts behavior on others, causing the agent to skip a valid action or stop searching too early. Object-type confusion failures ($4\%$) refer to cases where the agent confuses semantically similar objects. Together, these clusters show that ALFWorld failures mainly arise from search efficiency, over-constrained prompting, and object-specific grounding errors.

Panel (b) shows that prompt dominance scales inversely with base-model strength: a prompt rule yields gains only for models that reliably follow it. Sonnet, being the strongest base, derives nearly all improvement from prompt edits alone; search-order heuristics in the system prompt suffice because it consistently obeys them. GPT-5.4 supplements prompts with a processor (introduced at round three) that manages its step budget for transformation tasks. Qwen3.5 requires the most varied mix (prompt, config, and processor), including a processor that intercepts its reasoning text and re-emits tool calls when needed, a mechanical fix for a failure that prompt-level steering cannot resolve. The shared pattern is prompt-first, with structural levers recruited only when prompts prove insufficient. The weaker the base model, the sooner evolution falls back from prompt-based steering to config or processor enforcement, visible in the growing non-prompt segments from Sonnet to Qwen.

Panel (c) shows that evolution reduced the main ALFWorld failure clusters, with different levers mattering for different task agents. For Qwen3.5, processor and config edits achieved the strongest effects, with hit-rates of $0.84$ and $0.71$, respectively. These edits directly addressed mechanical failures by re-emitting missed tool calls and adjusting execution budgets, allowing Qwen3.5 to improve by $+44.0$pp and approach the closed-model runs. For Sonnet, the remaining failures were less structural, so prompt edits were sufficient for most gains, reaching a $0.49$ hit-rate, while the processor edit had only a marginal effect ($0.14$). Two clusters were only partially reduced: prompt-rule side effects, which were introduced by some evolved heuristics and then patched in later rounds, and long-path failures, where some episodes still exceeded the available interaction budget. Overall, ALFWorld shows a clear model-dependent pattern: stronger models benefit mainly from prompt-level steering, whereas weaker models require more structural support through processor and configuration edits.

Panel (a) summarizes the WebShop failure clusters accumulated across the run. Early failures are dominated by search and pagination loops, where the agent repeatedly reformulates queries or cycles through result pages without committing to a purchase. As evolution reduces these control-flow errors, the remaining failures shift toward product-selection judgment. The largest cluster, wrong product ($46\%$), occurs when the agent selects an item from the wrong category or settles on a weak match before comparing alternatives. Pagination loop failures ($21\%$) capture the remaining cases of repeated next/previous navigation without progress. Colour matching failures ($17\%$) arise when the agent mishandles shade equivalence or site-specific color labels, such as treating “wine” and “red” as incompatible. Attribute check failures ($17\%$) occur when the selected item is close to the request but fails on a required detail, such as size, sleeve length, or another unverified attribute. Overall, the cluster shift indicates that evolution first reduces navigation loops, after which the main errors concentrate in product matching and attribute verification.

Panel (b) shows that prompt edits drive most of the improvement across all three models, with processor edits serving as a consistent secondary lever. This pattern matches WebShop’s main control-flow failures. Prompt rules help the agent search more efficiently and commit earlier, while advisory processors reinforce these rules during execution by adding warnings when the agent begins to repeat searches or cycle through pagination. For product-selection failures, evolution introduces more targeted support: a colour-matching tool helps resolve shade-equivalence cases, and config edits help weaker models maintain context over longer shopping sessions. Overall, WebShop requires a mixed response: prompts improve high-level shopping strategy, processors reduce navigation loops, tools support attribute matching, and config edits stabilize long-session behavior.

Panel (c) shows that evolution partially reduced the WebShop failure clusters. Prompt edits were the most consistently effective lever across models, with hit-rates of $0.37$–$0.50$, while config edits helped the two weaker models maintain context over longer sessions. These changes reduced early search and pagination loops, raising performance from $60\%$ to a peak of $76\%$. The remaining clusters proved harder to close. Advisory processors produced only modest gains ($0.20$–$0.25$), so some pagination failures persisted. The colour-matching tool did not improve performance in this run ($0.0$ hit-rate), leaving the colour-matching cluster largely unchanged. Overall, WebShop benefits most from prompt and config edits, while residual navigation loops and product-judgment errors remain the main sources of instability.

Excluding harness-interrupt traces, the failures are judgment-heavy: the top two clusters, premature / unverified action ($28\%$; committing a booking, refund, or device fix before a precondition holds) and wrong selection / count ($24\%$), together exceed half and concern when to commit and what to pick rather than mechanical execution. The remainder is procedural (incomplete multi-step fix $16\%$, missed step / sub-task $14\%$) or policy-related (misinterpretation $13\%$). The smallest cluster, capability-boundary confusion ($5\%$), is $\tau^{3}$-specific: some telecom faults live on the user’s handset, where the agent has no device-side tool and the failure is its treating that boundary as a missing capability.

Evolution is prompt-and-processor driven for every model, with zero tools edits: the tool set is fixed and no cluster is one that a new tool could close. Sonnet 4.6 splits prompt/processor ($23/18$), GPT-5.4 ships the most balanced mix ($19/20$), and Qwen3.5-9B ships fewer overall ($14/9$). Since $\tau^{3}$ failures are control-flow and judgment errors, the productive levers are prompt rules that encode the policy’s ordering constraints and processors that enforce them mid-dialogue.

Config is the sharpest lever where used (Qwen3.5 $0.67$, GPT-5.4 $0.33$ hit-rate) but is shipped rarely; the high-volume prompt and processor levers are moderately effective ($0.27$–$0.35$), matching the control-flow nature of the dominant clusters. Gains track base-model headroom: GPT-5.4 starts lowest ($76.2\%$) and gains most ($+14.5$pp), Sonnet 4.6 gains $+5.4$pp, and near-ceiling Qwen3.5-9B only $+1.1$pp. The loop is not monotone; Sonnet’s telecom run reaches $100\%$ at R4, regresses to $80.7\%$ at R7 after a sixth consecutive same-bucket edit, then recovers to $99.1\%$ by R9 (Section 6.6). Overall, the ordering-enforcing levers close the premature-action and missed-step clusters most reliably, while wrong-selection and policy-judgment errors are the harder residual.

Panel (a) summarizes the failure clusters pooled across all rounds and all three models. The dominant cluster is incomplete fix ($62\%$), where the agent reaches the right region and produces a valid patch but covers only one branch or call site while the gold patch needs several. Wrong diagnosis ($19\%$) follows, covering edits to the wrong file or abstraction level after misreading the root cause. The remaining tail is mechanical rather than cognitive: no edit attempted ($6\%$), Edit anchor mismatch ($5\%$), and budget exhausted ($4\%$). Notably the composition is the inverse of reward-hacking: failures are under-fixes, not gamed evaluations, because the harness applies the gold test patch before the model patch and blocks test-file writes.

Panel (b) shows that SWE-bench is prompt-first for every model, with the secondary lever tracking base-model strength. All three runs ship zero tools edits, since unlike GAIA the failure set has no mechanical-retrieval cluster a tool could close. Sonnet pairs prompt with an equal share of processor edits ($7$ each), using workflow nudges to shape an already-competent coder; GPT-5.4 leans hardest on prompt ($8$ ships) and uses config ($4$) to revert a harmful nudge and restructure its strategy phases; Qwen3.5 spreads its few ships across prompt, processor, and config ($6/3/3$). The shared logic is prompt-first, with structural levers recruited as the base model weakens.

Panel (c) reveals a sharp capability floor. For the strong models the productive levers genuinely close failures: GPT-5.4’s config edits reach $0.48$ and prompt $0.39$, while Sonnet’s prompt and processor levers both land at $0.40$. For Qwen3.5-9B every lever collapses to near-zero (prompt $0.05$, config $0.05$, processor $0.06$), an order of magnitude lower, because the $9$B base cannot execute the predicted fixes. The same loop that lifts GPT-5.4 from $45\%$ to a $64\%$ peak and stabilizes Sonnet near $87\%$ yields only noise on Qwen3.5 (peak $42\%$, zero durable gains). Overall SWE-bench improves through prompt edits that broaden fix scope and config edits that restore workflow pacing, but only for models strong enough to act on them.