Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution

The system under test (SUT) is the native harness, which manages model interactions, tool coordination, lifecycle behavior, and permissions. The model, task, environment, and resource budgets are controlled inputs. The evaluator and the capture machinery are measurement apparatus. Apparatus failures are therefore not counted as harness defects.

We capitalize Harness layer and Agent layer when referring to the two score sheets, and use Judge for the configured semantic grader. We distinguish four claim levels. A criterion verdict records whether the selected evidence satisfies a declared scoring condition. A confirmed Harness defect additionally reconciles a failed Harness criterion with the question-specific authoritative source. An activated candidate mechanism link records that a predeclared Harness–Agent endpoint pair and its activation condition match the run. A validated propagation claim additionally requires confirmation of both endpoint verdicts and manual review of event order, an observable intermediate distortion, and plausible competing explanations. This hierarchy keeps operational scores and propagation claims auditable without treating an internal runtime label as a causal conclusion.

For harness $h$, model $m$, task $t$, and execution instance $r$, one run is defined as the ordered record

where $\tau$ is the normalized event sequence, $\Delta W$ is the workspace change relative to the frozen baseline, $G$ is the apparatus-admission state, $V_{H}$ and $V_{A}$ are criterion verdicts for the Harness and Agent layers, and $L_{\mathrm{act}}$ is the set of activated candidate mechanism links. The index $r$ distinguishes execution instances; the reported population retains one selected execution per cell. Run identity additionally binds the task source, resolved model and harness configuration, container image, execution-instance identifier, archive manifest, and artifact digests.

A nominal “tool result” may have several non-equivalent representations: the tool or process produces an external result; the harness may transform it; the model receives a model-facing payload; a native session archive records some representation; and the evaluator observes a normalized event. TraceTrial- Bench consequently uses question-specific authorities. Environment or simulator state is authoritative for what actually happened, changed artifacts are authoritative for final persisted artifact state, provider-request evidence is authoritative only for the structured tool declarations it retained, and a native model-facing result is authoritative for what the model could observe.

The implementation has two scored layers after apparatus admission. The Agent layer measures task actions, workspace state, and deliverables, whereas the Harness layer targets native harness behavior using trajectory and probe evidence. The two verdict vectors remain distinct; the released protocol contains one explicitly outcome-conditioned applicability state, described in Table 4. Container lifecycle, archive integrity, trace scoreability, probe integrity, and private-oracle availability are verified before semantic scoring is admitted. Apparatus admission is not a third score and cannot compensate for either scored layer. Figure 3 shows the boundary and the resulting dataflow.

The formal evaluation organizes Harness reliability into seven top-level dimensions: Context (C), Execution (E), Governance (G), Lifecycle (L), Observability (O), Tools (T), and Verification (V) (li2026agent). The dimensions are defined by the Harness obligation under test rather than by the application scenario in which it is exercised. Figure 4 shows the property distribution (left) and summarizes the scope of each dimension and its task coverage in the 95-task evaluation population (right).

Cross-cutting behaviors are assigned according to the property being tested. For example, state isolation is evaluated under Context when the question is whether information crosses a session boundary, whereas child-agent completion is evaluated under Lifecycle when the question concerns creation, delivery, or termination. A mechanism is included only when its trigger, expected behavior, relevant observation boundary, and verdict conditions can be specified before execution. Coverage is then tracked separately for task assignment, native capability availability, mechanism activation, and evidence sufficiency. Integrity of the measurement apparatus is assessed by the admission gate and is therefore outside the seven-dimension taxonomy.

Each task starts from a taxonomy subdimension and identifies the native mechanism, the property under test, the model-visible consequence, and an external truth anchor. The scenario then exercises that mechanism through ordinary domain work. Each task pairs Harness-layer criteria with Agent-layer deliverables and uses deterministic state or artifact checks wherever an oracle is available.

A task package fixes the prompt or turn sequence, initial workspace, allowed tools, budgets, expected outputs, deterministic verifier, two score sheets, and optional simulator or execution/security probes. Private simulator state and persisted baseline snapshots remain outside the Agent’s writable workspace. Review checks prompt solvability, mechanism reachability within budget, verifier agreement with task truth, hidden leakage, and equivalent domain semantics across eligible harnesses. The resulting 95-task population is the formal evaluation set.

The release includes 46 single-turn, 45 fixed multi-turn, and 4 simulator-driven tasks. Task triggers include oversized outputs requiring recovery, nonzero exits paired with success-shaped artifacts, pending jobs, facts crossing session boundaries, and private values available only through an interaction channel. These conditions make the obligation observable while retaining a practical task and an independent truth anchor.

The release evaluates Codex, Claude Code, OpenClaw, and DeepSeek Harness through source-specific adapters. The runner freezes the task revision, harness adapter, requested and resolved model, reasoning setting, timeout, container image, judge model, and resource budgets. Each adapter launches the native harness and translates its observable records without replacing native prompt construction, tool transport, context policy, permissions, or session behavior. The task contract and oracle remain fixed while each harness retains its native execution strategy.

Semantic equivalence permits representation normalization but prohibits changes that introduce a capability, strengthen model-visible evidence, or alter task pass/fail conditions. The release manifest records the selected population and provenance of every run. Execution status, trace completeness, evaluation status, score, and attribution remain separate fields.

Coverage is tracked separately for task assignment, native capability availability, mechanism activation, and evidence sufficiency. A task can be present in the manifest without every selected execution exercising its target path. An unavailable named tool is not automatically a defect if the native harness supplies an equivalent usable path. If an adapter cannot preserve the task’s domain semantics or required evidence, the corresponding comparison is non-comparable under the prospective protocol. The current experiment’s full-grid inclusion policy is stated separately in Section 4.3.

Runs execute in task-scoped containers with a fixed container-internal workspace. The runner initializes a frozen task state, captures a private workspace baseline, and provisions deterministic task tools or stateful simulator services when declared. Task metadata controls user turns, tool calls, model calls, wall time, process containment, and other relevant resource bounds. The resolved configuration records the effective values.

The measurement apparatus is isolated from the Agent’s writable output surface. Runner-owned lifecycle records, archive manifests, provider-request metadata, private simulator ledgers, baselines, and execution/security probes carry explicit provenance and digest bindings. The release manifest serves as the authoritative index connecting task, configuration, run identity, evidence, and checksums.

The full trace is too large and too heterogeneous to place in every semantic Judge call. TraceTrial-Bench therefore separates archival evidence from criterion evidence. Archival evidence is retained for audit. A criterion receives a bounded packet produced by a declared selector in its score sheet.

A run is scoreable only when its result, resolved configuration, task, model, and normalized harness identity agree. Lifecycle and cleanup records must be complete, and the trace and summary must be covered by the archive manifest. All task-declared baselines, probes, budgets, and private-oracle records must also pass their integrity checks. A structured failed trace is admissible only for a declared task in which the observed lifecycle failure is itself the target property. Execution, evaluation, and infrastructure states remain separate in the run record. Admission establishes structural integrity and source provenance. Criterion evaluation then records selector coverage and the exact evidence presented for judgment.

Evaluation distinguishes criteria that can be decided by a task-specific verifier from those that require semantic interpretation. Each task therefore defines separate Harness- and Agent-layer score sheets, and designates every criterion as either a deterministic rule or a semantic rubric. Before any Judge call, the deterministic outputs are checked against the declared criterion identifiers and fixed. This preserves objective task checks while restricting semantic judgment to criteria for which a deterministic oracle is not available.

Semantic criteria are evaluated individually from their criterion-specific evidence packets. Agent rubrics are evaluated first, using changed artifacts as primary evidence. Once the complete Agent-layer result has been fixed, Harness rubrics are evaluated from trajectory and probe evidence. Agent verdicts are provided only as downstream context: they may describe the impact observed in the run, but they are not evidence that a Harness criterion failed. For every semantic criterion, the archive retains the matched packet together with the Judge prompt, response, parsed score, and rationale. If a required selector returns no evidence, the current implementation assigns zero and explicitly records the empty selection. This indicates that satisfaction was not established under the declared evidence contract and does not independently prove a Harness defect.

The two score sheets are distinct but not statistically independent because state 2 depends on the completed Agent outcome. Corpus-level reporting uses the finalized binary verdicts. Intermediate diagnostic codes retained in individual rubric artifacts are not analyzed as separate experimental outcome categories.

For layer $X\in\{H,A\}$ with $n_{X}$ finalized items and $p_{X}$ passed items, the reported score is

Items are unweighted within a layer. A run passes the combined evaluator only at $S=100$, but the continuous score, item verdicts, and attribution are all retained. Figure 7 summarizes the actual order.

Every released run is subjected to automated checks of run identity, archive integrity, and admission status. Corpus-level analysis reports criterion verdicts and automatically activated candidate links from the archived evidence. These are operational outputs: an activated link records agreement with its predeclared endpoints and activation condition, but does not by itself establish that the proposed mechanism propagated a failure.

Attribution begins with the criterion-level verdicts, but not every Harness-layer zero represents a failure attributable to the Harness. In some runs, an Agent action makes the target property unobservable. For example, a wrapper command may replace the exit status of the command under test with its own exit status. Let $F_{H}$ and $Z_{A}$ denote the raw sets of zero-scored Harness and Agent criteria, respectively, and let $B\subseteq F_{H}$ denote the Harness criteria affected by such an observation bypass. We define the Harness-side failure set used for attribution as

Here, $\setminus$ denotes set difference: a criterion belongs to $Z_{H}$ if and only if it is zero-scored at the Harness layer and is not marked as bypass-affected. Thus, $Z_{H}$ is obtained by removing every criterion in $B$ from the raw zero set $F_{H}$. Bypassed criteria remain visible as zeros in the score report, preserving the original evaluation outcome, but they do not contribute to $Z_{H}$. Membership in $Z_{H}$ is itself only an operational Harness-layer result, not a confirmed defect. The empirical results in Section 5 use the raw set $F_{H}$ without this adjustment, as specified in Section 4.3. The adjusted set and stored runtime labels remain available as separate implementation outputs. A criterion enters the confirmed set $D_{H}\subseteq Z_{H}$ only after its verdict has been reconciled with the evidence source appropriate to that criterion.

Before execution, a task may declare that failure of one Harness criterion can affect a particular Agent criterion through a specified intermediate mechanism. We represent such a declaration as

where $h_{\ell}$ and $a_{\ell}$ are the two criterion endpoints and $\mu_{\ell}$ describes the intervening information, action, or state distortion. The implementation also stores a qualitative descriptor for review prioritization, which is not interpreted as a calibrated probability. Task-specific declarations define the available endpoint pairs. Some tasks assess faithful reporting of model-visible information and do not declare a Harness-to-Agent endpoint pair. The experimental analysis counts activated pairs stored with final evaluations. It does not infer the size of the full candidate catalog from the subset of links that happened to activate.

At runtime, the task declarations and applicable compatibility rules form the candidate pool $\mathcal{C}^{\mathrm{run}}_{t}$. For a declared activation condition $q_{\ell}(R)$, the implementation constructs

Activation is computed from the raw Harness set $F_{H}$ before bypass adjustment, as in the released implementation. It establishes that the declared endpoints and activation condition are present in the run, but does not establish that the intermediate mechanism actually propagated the failure.

A candidate link is promoted to the validated set $L_{\mathrm{val}}\subseteq L_{\mathrm{act}}$ only after manual review confirms both endpoint verdicts, reconciles the Harness endpoint with its criterion-specific evidence source, and finds an observable intermediate distortion in an event order consistent with the declared mechanism. The review also considers provenance and plausible competing explanations. Hence the Harness endpoint of every validated link belongs to $D_{H}$, and only $L_{\mathrm{val}}$ supports a validated propagation claim.

The Agent failures covered by activated candidate links are

TraceTrial-Bench reports $|Z_{H}|$ independently and partitions $Z_{A}$ into linked and unlinked subsets. These counts are recoverable from the serialized verdicts, bypass annotations, and candidate links, allowing a run to contain both a Harness-side zero and an unrelated Agent-side zero. Table 5 presents seven ordered decision rules that yield six distinct stored labels, with two rules producing agent_caused.

The stored strings are retained for compatibility with the released runtime record and are interpreted conservatively in the paper. In particular, agent_caused does not assert that the Harness is defect-free when $Z_{H}$ is non-empty, and harness_caused denotes candidate-link activation rather than a causal conclusion. Confirmed defect absorption requires the relevant Harness endpoint to belong to $D_{H}$, and validated propagation requires $L_{\mathrm{val}}\neq\emptyset$.

For each run that passes admission and completes scoring, the archive preserves three classes of records: (i) configuration and provenance, including the resolved configuration, run result, and lifecycle and archive manifests; (ii) execution evidence, including normalized and native traces, provider-request metadata, applicable simulator, budget, and probe evidence, workspace changes, and persisted scoring inputs; and (iii) evaluation artifacts, including verifier outputs, criterion-specific evidence and Judge records, and the combined layered score. The release manifest associates each selected run with its task and resolved configuration and records the corresponding artifact paths and checksums. The serialized result contains the apparatus state, $V_{H}$, $V_{A}$, $L_{\mathrm{act}}$, and bypass annotations; together, these fields determine the two-axis summary without additional heuristic inference.

Let $\mathcal{R}_{A}$, $\mathcal{R}_{H}$, and $\mathcal{R}_{\mathrm{attr}}$ denote the reporting populations for Agent, Harness, and attribution metrics. In this study all three are the complete selected archive, as specified in Section 4.3. Agent-level performance is summarized by the task success rate

where a run is successful only if it contains no zero-scored Agent criterion. Harness-level performance is summarized by the micro- and macro-averaged criterion pass rates

The micro average assigns equal weight to each finalized Harness criterion, whereas the macro average assigns equal weight to each run. Because $p_{H}(R)$ is derived from finalized verdicts, both quantities incorporate the prescribed outcome-conditioned resolution of raw state 2 in Table 4. They therefore measure operational Harness-criterion satisfaction and should not be interpreted as confirmed- defect rates.

For $\mathcal{R}_{\mathrm{attr}}$, we report three complementary proportions: (i) runs with at least one activated candidate link ($L_{\mathrm{act}}\neq\emptyset$); (ii) runs with at least one raw Harness-layer zero but no Agent-layer zero ($F_{H}\neq\emptyset$ and $Z_{A}=\emptyset$); and (iii) runs containing at least one unlinked Agent zero in $Z_{A}\setminus Z_{A,\mathrm{linked}}$. Measurement-quality records—including admission outcomes, trace completeness, and selector match counts—remain separate from performance aggregates. The current analysis retains all finalized cells and does not estimate corpus-level selector recall, Judge agreement, or candidate-link precision.

The evaluated Harnesses are Claude Code, Codex, DeepSeek Harness, and OpenClaw. Source-specific adapters provision the task and archive native evidence while retaining each Harness’s execution and tool-use interfaces. The nine model groups are DeepSeek-V4-Pro, Gemini 3.7 Flash, GLM-5.3, GPT-5.6-Sol, Grok-4.6, HY3, Kimi K3, Opus 5, and Qwen3.8-Max. Reasoning settings follow the archived configurations. Original model identifiers, resolved configurations, and invocation fields remain in the run records, while the nine names above serve as the reporting groups used throughout the results.

The analysis covers 95 tasks, each evaluated across all 36 model–Harness combinations. Taxonomy assignments are recovered from the task paths recorded with the runs and normalized as described in Appendix : the four legacy ask-user tasks are assigned to Tools, while the task measuring truthful timeout and result handling is classified under Execution. The resulting population contains 15 Context, 27 Execution, 9 Governance, 19 Lifecycle, 7 Observability, 17 Tools, and 1 Verification task, covering 40 mechanism-level properties. The generated task ledger records the exact membership and metadata sources used in this analysis.

The data freeze contains

There are 855 cells per Harness and 380 per model. The unit of analysis is the root task–model–Harness run directory in outputs/traces. Nested result.json files emitted by tools or simulators are not additional experimental runs. Every selected cell has a final combined_score.json, and no model block is pending and no missing score is imputed. The archive contains selected outcomes, including replacement executions from earlier recovery, rather than independent repetitions per cell. We therefore describe the observed matrix and do not estimate within-cell repeatability.

Runs execute in Docker with the task workspace mounted at /workspace. Resolved configurations and container records retain the Harness invocation and image identity. Task packages define the prompt or turn sequence, fixture, tools, expected deliverables, verifier, and resource limits. The recorded effective wall-time limits span 120–5,400 seconds. The corresponding task definitions span 1–120 user-input turns and 5–300 task-tool calls. These are task-specific ceilings, not observed turn or call counts. The per-run resolved limit, rather than an example command or directory name, identifies the applicable execution budget.

Task scheduling and recovery conditions vary across the collected runs. The present analysis consequently compares finalized functional outcomes and criterion verdicts. It does not interpret duration differences as a controlled throughput experiment. Task-specific revisions and recovery runs also limit causal interpretations of a difference between Harnesses.

All 3,420 archived final evaluations use deepseek-v4-flash as the separate semantic Judge. The stored per-criterion output limits are 4,096 tokens in 1,966 evaluations, 8,192 in 1,452, and 32,000 in two. Thus Judge identity is fixed, while the archived scoring budget is not identical across all cells. We preserve these final verdicts and report their configuration rather than treating the collection as a uniform Judge-budget experiment.

Agent- and Harness-layer items, their binary verdicts, and activated candidate links are read from artifacts/llm_judge/combined_score.json. The root result.json supplies execution identity and historical metadata, but its score field is not the final-score authority. Execution status, evaluation status, and final criterion verdicts remain separate records. We verify that every selected combined score has the expected task identity, both nonempty layer verdict vectors, binary item values, and a total consistent with the 50/50 formula in Equation 3.

All 3,420 final evaluations are included exactly once using their stored criterion values. Execution-state flags and auxiliary annotations do not affect inclusion or scores.

For run $r$, let $p_{A,r}$ and $n_{A,r}$ be the passed and total Agent criteria, and define $p_{H,r}$ and $n_{H,r}$ analogously. Agent task success requires $p_{A,r}=n_{A,r}$. Agent Score and Harness Macro are the means of the per-run fractions $p_{A,r}/n_{A,r}$ and $p_{H,r}/n_{H,r}$. Harness Micro pools passed and total criteria across runs: $\sum_{r}p_{H,r}/\sum_{r}n_{H,r}$. All reported rates are multiplied by 100. The Layered score averages the two per-run layer scores with equal weight. A combined pass requires every criterion in both layers to pass; it must not be confused with Agent-only task success.

Every Harness is evaluated on the same 855 model–task cells. Fixed-model comparisons use all 95 tasks for each Harness; model averages use all four Harnesses and 95 tasks. Dimension-level summaries count a task once under its normalized primary dimension. Since criterion counts vary by task, we report both micro and macro Harness rates and make the task population explicit.

The empirical contingency uses the raw sets $F_{H}=\operatorname{fail}(V_{H})$ and $Z_{A}=\operatorname{fail}(V_{A})$. All zero-scored items contribute normally, and auxiliary bypass annotations do not subtract items from $F_{H}$. This is the reporting convention for the current experiments, distinct from the optional adjusted Harness set $Z_{H}$ defined in Section 3.5. A jointly clean run has $F_{H}=Z_{A}=\emptyset$. An operational absorption state has $F_{H}\neq\emptyset$ and $Z_{A}=\emptyset$. Crossing the empty/nonempty status of the two sets yields four mutually exclusive states. Their proportions use all 855 runs per Harness and sum to 100%. A shortfall denotes at least one failed criterion, not a runtime execution error.

Candidate-link counts come from the stored endpoint pairs after final scoring. We check that each stored link connects a failed raw Harness item to a failed Agent item. A run can contain both linked and unlinked Agent failures, so these two evidence categories can overlap. Neither co-occurring zeros nor a stored link establishes a confirmed Harness defect or causal propagation. Runtime class labels are retained in the analysis ledger, but the contingency is reconstructed directly from item verdicts to maintain the declared raw-score convention.

For primary dimension $d$, let $J_{d}$ contain the runs with at least one failed criterion in both layers, and let $L_{d}\subseteq J_{d}$ contain those with at least one stored candidate link. We report $100|L_{d}|/|J_{d}|$. Each run contributes once regardless of its number of failed items or links. The pooled rate is $100\sum_{d}|L_{d}|/\sum_{d}|J_{d}|$, not an unweighted mean of the dimension rates. This statistic measures recorded link activation among joint failures.

Resource columns report mean recorded usage per run: input, output, and total Tokens in thousands, and estimated Agent cost in USD. We apply the same model-specific prices published on the providers’ official websites across all Harnesses. Input cost partitions ordinary input, cache reads, and cache writes using the available counters. Unspecified cache components use the ordinary input rate. Output cost includes recorded reasoning Tokens without counting them again. Total cost is input plus output cost, averaged over runs with valid recorded usage. These are standardized Token-cost estimates, not historical invoices. Judge calls, discarded executions, payment fees, storage, and tool charges are outside their scope.