EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings

Shiva Krishna Reddy Malay, Shravan Nayak, Jishnu Nair, Sagar Davasam, Aman Tiwari, Sathwik Tejaswi, Sridhar Krishna Nemala, Srinivas Sunkara

and 1 more authors

Sai Rajeswar

arXiv:2603.13594v1 · This workspace stays on this version. Check arXiv for newer versions

Affiliations and author notes

ServiceNow Research
shivakrishnareddy.ma@servicenow.com
Mila - Quebec AI Institute
Université de Montréal

Some source details need review 7 to check

Original formulas, table values and figures are preserved. Comparison can miss differences; use the original PDF to check details.

Reading view checked: 95 math expressions, 0 rendering failures; 25 of 25 images available.

Compared 93 of 109 source passages against 28 of 32 PDF pages.

A3_p3_pic1: the arXiv HTML draws 1 text or formula block(s) of this graphic outside its frame; each was scaled into its frame (to 95% or more) without changing any text or formula. Text may look smaller than in the PDF.
This long paper was only partially compared; the original sources remain available.
An unsupported comparison claim was rejected; source verification is incomplete.

HTML (S1_p1_1) says “Anthropic, 2025c”; the original PDF (page 1) says “Anthropic, 2025b.” Check PDF page 1

The PDF page 7 image reads “Anthropic, 2025b” for Claude Opus 4.5, while HTML block S4_T2 reads “Anthropic, 2025c”; the PDF contradicts the HTML. Check PDF page 7

HTML (S4_SS1_p1_1) says “Anthropic, 2025c; Anthropic, 2025a”; the original PDF (page 6) visibly reads “Anthropic, 2025b;a,” contradicting the HTML. Check PDF page 6

HTML block S4_SS3_SSS0_Px1_p1_1 says “Overall, Claude Opus 4.5 achieves the best average task completion (37.4%)” but provides no citation year. PDF page 6 reads “Claude 4.5 (Anthropic, 2025b;a) variants”; the supplied HTML evidence does not establish 2025c. Check PDF page 6

25 of 25 figure, table and graphic images captured. Open original PDF

Abstract

Large language models are shifting from passive information providers to active agents intended for complex workflows. However, their deployment as reliable AI workers in enterprise is stalled by benchmarks that fail to capture the intricacies of professional environments, specifically, the need for long-horizon planning amidst persistent state changes and strict access protocols. In this work, we introduce EnterpriseOps-Gym, a benchmark designed to evaluate agentic planning in realistic enterprise settings. Specifically, EnterpriseOps-Gym features a containerized sandbox with 164 database tables and 512 functional tools to mimic real-world search friction. Within this environment, agents are evaluated on 1,150 expert-curated tasks across eight mission-critical verticals (including Customer Service, HR, and IT). Our evaluation of 14 frontier models reveals critical limitations in state-of-the-art models: the top-performing Claude Opus 4.5 achieves only 37.4% success. Further analysis shows that providing oracle human plans improves performance by 14–35 percentage points, pinpointing strategic reasoning as the primary bottleneck. Additionally, agents frequently fail to refuse infeasible tasks (best model achieves 53.9%), leading to unintended and potentially harmful side effects. Our findings underscore that current agents are not yet ready for autonomous enterprise deployment. More broadly, EnterpriseOps-Gym provides a concrete testbed to advance the robustness of agentic planning in professional workflows.

https://enterpriseops-gym.github.io

Graphic from the paper; its text follows

Figure 1: Performance–cost tradeoff for agentic tool use on EnterpriseOps-Gym We plot task success rate against estimated cost per task for both closed-source and open-source models. Open-source models incur lower cost but achieve consistently lower success rates. While higher-cost models offer modest performance gains, they remain far below reliable task completion.

Figure 2: Overview of EnterpriseOps-Gym: A benchmark for stateful agentic planning and tool use. (Top-left) EnterpriseOps-Gym spans eight enterprise domains and evaluates multi-step agentic planning, policy adherence, state-driven tool calling and cross domain orchestration in a reproducible sandbox. (Top-right) Domain experts create sandox and author realistic single- and cross-domain tasks, execute ground-truth trajectories, and write outcome-based verification logic with multi-stage quality assurance along with a human written oracle plan for completing the task. (Bottom) Given a task and constraints (as system level policies), agents interact with the environment and execute tools. They are evaluated by final-state verifiers that check goal completion, policy compliance, and side effects. In the above example the agent fails to adhere to system policy for linking case knowledge which mandates setting a parameter to “suggested” when a knowledge base is automatically discovered. Furthermore, the agent fails to properly send an email notification due to an unresolved identifier for the given case.

LLMs today are most commonly deployed as conversational assistants, answering questions, drafting emails, and summarizing documents (OpenAI, 2025; Anthropic, 2025c). But a far more consequential capability is rapidly emerging, namely LLMs as autonomous agents that act on behalf of users (Xu et al., 2024). Consider an agent that, from a single instruction, searches the web for real-time inventory data, identifies the best product, and executes the purchase, all without further input. Recent advances in planning and tool use have made the vision of an AI worker increasingly plausible: autonomous agents handling professional workflows from software engineering (Jimenez et al., 2024) and data analysis (Drouin et al., 2024) to sales operations and enterprise administration (Huang et al., 2025a; Huang et al., 2025b). Yet to function effectively in real-world professional deployments, such agents must do more than follow instructions. They must (1) maintain state coherently across long sequences of interleaved tool calls, (2) execute multi-step plans spanning dozens of actions, and (3) strictly adhere to the access control policies and procedural rules that govern the workplace.

These requirements are especially relevant within the enterprise domains. Here, agents do not merely retrieve information; they directly modify live databases, trigger downstream workflows, and affect real users. Actions are stateful and often irreversible, errors propagate silently across interconnected systems, and strict organizational policies constrain every step. Figure 2 shows one such example task, where the interplay of the user task and the nuanced system policy constraints demands more than surface-level instruction following. The agent must not only execute a multi-step workflow — searching a case, retrieving a KB article, and notifying the customer — but also respect a three-tiered policy governing how KB articles are tagged, and independently resolve a hidden piece of information (the customer’s email) by chaining through multiple tools. Equally important, agents must also know when to stop. Blindly attempting a policy-violating task corrupts system state, posing a direct safety risk in production environments.

Despite the urgency of this challenge, existing benchmarks fall short of capturing it. General tool-use evaluations (Li et al., 2023; Qin et al., 2024; Chen et al., 2025) treat tool calls as atomic and stateless, measuring accuracy on short sequences without state dependencies or cross-system coordination. Enterprise-focused benchmarks have emerged (Drouin et al., 2024; Boisvert et al., 2024; Huang et al., 2025a; Huang et al., 2025b; Jha et al., 2025; Vishwakarma et al., 2025) to address challenges in professional environments. While they make important contributions, they are typically confined to a single vendor ecosystem (e.g., Salesforce or ServiceNow alone), with shallow environments of fewer than 25 database tables, under 50 tools, short task horizons, and limited policy constraints governing agent behavior (see Table 1).

To bridge this gap, we introduce EnterpriseOps-Gym, a benchmark designed to evaluate agentic planning within a high-fidelity enterprise simulation. EnterpriseOps-Gym spans eight interconnected ecosystems, ranging from general productivity tools like Calendar, Drive, Teams and Email to mission-critical business functions like Human Resource (HR), IT Service Management (ITSM), Customer Service Management (CSM), unified by a Hybrid category demanding coordinated cross-domain execution. The benchmark comprises 1,150 expert-authored tasks across these domains, including 30 infeasible scenarios that test whether agents correctly refuse unsatisfiable requests without leaving side effects on the system. Tasks are verified by hand-written SQL scripts that check goal completion, state integrity, policy compliance, and unintended side effects. EnterpriseOps-Gym runs on a fully interactive, containerized environment hosting 164 relational database tables and 512 functional tools, with expert trajectories averaging 9 steps and reaching up to 34. Together, this scale is designed to stress-test the core challenges of enterprise automation: long-horizon planning, cross-system state management, and policy-constrained execution.

Our evaluation of 14 frontier models reveals a significant gap between current agentic capabilities and enterprise requirements. We find that performance is strongly shaped by domain complexity. Models fare best on collaboration tasks, with top models reaching 51–52% on Email, Teams, and Drive, but drop sharply on policy-governed domains such as ITSM (28.5%) and cross-domain Hybrid tasks (30.7%). These are precisely the domains where constraint-aware reasoning is unavoidable. The best overall model, Claude Opus 4.5, achieves only 37.4%, with open-source models lagging further behind. Infeasibility detection is a critical weak point, with even the best model refusing policy-violating tasks cleanly only 53.9% of the time. Test-time compute scaling helps but is not a universal remedy, with some workflows plateauing early or showing limited improvements regardless of thinking budget. Importantly, we identify strategic planning and not tool use as the primary bottleneck. Adding distractor tools has negligible impact, while providing human-authored plans improves models by 14–35 percentage points. More complex multi-agent orchestration does not close this gap either; decomposing tasks into subtasks can even regress performance due to strong sequential state dependencies. Together, these results underscore that current agents are not yet ready for autonomous enterprise deployment.

Overall, our contributions are as follows:

•
We introduce EnterpriseOps-Gym, a benchmark of 1,150 expert-curated tasks across eight enterprise domains with outcome-based verification enforcing goal completion, state integrity, policy compliance, and side-effect checks, including 30 infeasible tasks designed to evaluate safe refusal behavior.
•
We develop a fully interactive, containerized enterprise environment with 164 relational database tables and 512 functional tools, an order of magnitude more complex than prior enterprise benchmarks.
•
We evaluate 14 frontier models, uncovering and analyzing systematic failure patterns across planning, state management, and policy compliance, and provide actionable insights to build more reliable enterprise agents. We also release EnterpriseOps-Gym to the community to advance research in stateful agentic planning and enterprise tool use.

Original Table 1 — Table 1: Comparison with existing agentic benchmarks. DB Tables reports the number of unique database tables in the environment; Avg. FK measures average foreign keys per table, indicating relational density and the complexity of inter-table dependencies agents must navigate. — denotes values not reported in the original work. ∗Avg. Steps reflects ideal human-authored execution trajectories; model trajectories may require significantly more steps. Human Plans refer to step-by-step natural language plans written by experts to complete the task. †Parenthetical values indicate the number of unique task templates.

Table 1
Benchmark	Focus	Num.	Num.	Num.	Avg.	DB	Avg.	Refusal	Human Task	Human
Domains	Tasks	Tools	Steps	Tables	FK	Ability?	Curation?	Plans?
General Tool Use
API-Bank (Li et al., 2023)	Tool-use	8	314	73	3	0	0	✗	✓	✗
ACEBench (Chen et al., 2025)	Tool-use	8	2000	4538	2	0	0	✓	✓	✗
$\tau$ -bench (Yao et al., 2024)	User Interaction	2	165	28	—	3	0.7	✓	✓	✗
$\tau^{2}$ -bench (Barres et al., 2025)	User Interaction	3	279	56	—	9	—	✓	✗	✗
Enterprise Specific
WorkArena (Drouin et al., 2024)	ServiceNow	7	33	30	10	7	0.9	✗	✗	✗
WorkArena++ (Boisvert et al., 2024)	ServiceNow	7	682 (†341)	30	30-50	7	—	✓	✗	✗
ITBench (Jha et al., 2025)	IT	3	94	10	—	—	—	✗	✓	✗
WorkBench (Styles et al., 2024)	Workplace	5	690 (†69)	26	2	5	0	✗	✓	✗
TheAgentCompany (Xu et al., 2024)	Startup	7	175	—	—	0	0	✗	✓	✗
CRMArena (Huang et al., 2025a)	Salesforce	1	1170 (†9)	27	—	16	1.3	✗	✓	✗
CRMArena-Pro (Huang et al., 2025b)	Salesforce	3	8560 (†19)	—	—	25	—	✓	✓	✗
EnterpriseBench (Vishwakarma et al., 2025)	Enterprise	5	500	46	3	17	1.2	✓	✗	✗
EnterpriseOps-Gym (Ours)	Enterprise	8	1150	512	9∗	164	1.7	✓	✓	✓

LLM agent benchmarks broadly fall into three thematic groups: general-purpose tool-use and API evaluation, enterprise platform simulation, and agentic planning and computer-use. Table 1 summarizes how EnterpriseOps-Gym compares across key dimensions.

Early benchmarking efforts focused on testing agents’ ability to call APIs and chain tools in general, open-domain settings. ToolLLM (Qin et al., 2024) establishes large-scale evaluation across over 16,000 real-world web APIs, while API-Bank (Li et al., 2023) provides a smaller, runnable system for assessing planning and retrieval across 73 tools. ACEBench (Chen et al., 2025) scales this further with 4,538 tools and dynamic multi-turn evaluation, and $\tau$ -bench (Yao et al., 2024) and $\tau^{2}$ -bench (Barres et al., 2025) introduce user simulation and refusal robustness as evaluation axes. These benchmarks make important contributions to measuring tool-calling accuracy but remain anchored to general web-oriented or open-domain APIs. They do not model the multi-system, policy-constrained, stateful tool ecosystems that characterize enterprise environments, which is the setting EnterpriseOps-Gym specifically targets.

A second class of benchmarks targets enterprise platforms directly. Single-platform environments restrict scope to one vendor: WorkArena (Drouin et al., 2024) and WorkArena++ (Boisvert et al., 2024) evaluate compositional task completion within ServiceNow, CRMArena (Huang et al., 2025a) and CRMArena-Pro (Huang et al., 2025b) focus on Salesforce specific workflows, and ITBench (Jha et al., 2025) targets IT incident resolution. Multi-domain efforts broaden the scope but with different emphases: TheAgentCompany (Xu et al., 2024) simulates a startup software company requiring agents to interact via web browser, bash terminal, and code execution, a paradigm fundamentally different from the structured tool-calling API setting of EnterpriseOps-Gym; WorkBench (Styles et al., 2024) covers office productivity tools; and EnterpriseBench (Vishwakarma et al., 2025) evaluates function-calling on sandboxed enterprise data. Across these benchmarks, relational database complexity is consistently limited to fewer than 25 tables, and tasks are either confined to a single vendor ecosystem or lack expert-authored grounding or explicit evaluation of refusal behavior. EnterpriseOps-Gym addresses these limitations by spanning eight enterprise domains across both operational systems (CSM, ITSM, HR) and collaboration services (Email, Calendar, Teams, Drive), with 164 database tables with high connectivity, 512 tools, 1,150 expert-curated tasks including policy-constrained infeasible scenarios.

Planning is a core capability for autonomous agents, and several benchmarks have studied it across diverse general-purpose settings. UserBench (Qian et al., 2025) evaluates agents on multi-turn travel planning tasks where simulated users express preferences incrementally and implicitly, requiring proactive intent elicitation. Gaia2 (Froger et al., 2025) evaluates agents in an asynchronous simulated mobile environment, testing search, execution, temporal reasoning, adaptability, ambiguity handling, and multi-agent collaboration. VitaBench (He et al., 2025) evaluates agents on complex multi-turn tasks drawn from real-world services like food delivery, in-store dining, and online travel using a simulated user with dynamic preferences. A parallel line of work focuses on computer-use agents: OSWorld (Xie and others, 2024), WindowsAgentArena (Bonatti et al., 2025), WebArena (Zhou et al., 2024) and UI-Vision (Nayak et al., 2025) benchmark agents on desktop and web GUIs requiring complex sequential decision-making. While these settings demand sophisticated planning, they operate in everyday scenarios or computing environments rather than the policy-governed, multi-system contexts of enterprise environments. EnterpriseOps-Gym shares the emphasis on extended planning horizons but is uniquely positioned at the intersection of planning depth and enterprise domain fidelity.

In this section, we describe the design and construction of EnterpriseOps-Gym, covering domain selection, sandbox environment, task formulation in Section 3.1, and dataset statistics in Section 3.2.

Figure 3: Task distribution across eight EnterpriseOps-Gym domains.

We selected domains based on three principles, in consultation with SMEs who have hands-on experience in enterprise software: (i) relevance to real-world industry verticals, (ii) diversity in policy complexity and data sensitivity, and (iii) availability of domain experts to author and validate authentic tasks. This led us to two complementary groups of domains.

The first group—Customer Service Management (CSM), Human Resources (HR), and Information Technology Service Management (ITSM)—represents the operational backbone of enterprise organizations. These domains are present in virtually every industry vertical and are characterized by strict process compliance, access control policies, and high-stakes workflows where errors have tangible consequences. CSM involves managing the full lifecycle of support tickets and service agreements; HR handles sensitive employee data under strict privacy and procedural rules; and ITSM covers backend IT operations including incident management and system configuration. Their policy-heavy nature and realistic complexity make them ideal for stress-testing constraint-aware planning.

The second group—Email, Calendar, Teams, and Drive—encompasses the universal collaboration tools used daily across all enterprise organizations. Agents must be proficient in these to function as effective AI workers. While individually simpler, these domains require sophisticated orchestration: Email handles complex mailbox workflows, Calendar manages time and resource access policies, Teams administers collaborative workspaces, and Drive governs file system integrity and security.

Finally, Hybrid mandates cross-domain orchestration across these fragmented tools, requiring agents to maintain context and data integrity while switching between systems. Together, the eight domains span the full enterprise workflow spectrum, from internal operations to customer-facing processes to collaboration infrastructure, enabling evaluation of both specialized and general-purpose agentic capabilities. Refer to Appendix B for more details on each domain.

We partnered with a professional data annotation firm (Turing $^{1}$ ) to assemble a team of over 160 contributors, including software engineers for building environments and Subject Matter Experts (SMEs) for technical domains like CSM, ITSM, and HR (refer to Section A.1). To provide a reproducible and realistic evaluation ground, we developed a containerized docker sandbox hosting domain-specific databases, APIs, and the tool execution layer. This environment mirrors enterprise constraints without exposing proprietary infrastructure. For each domain, we created a realistic database schema and populated it with seed data and tools to access and manipulate data. The underlying data is designed with a strong real-world perspective by SMEs. Guided by official product documentation and SME insights, we model realistic table structures, constraints and tools based on industry standard database schemas. Starting from a fixed seed of data and tools, annotators extend the environment with new tables, schemas, and records as each task demands. Refer to Section A.4 for more details on the environment.

The task creation process follows a rigorous pipeline designed to ensure high complexity and faithfulness to real-world workflows. In Scenario Design, annotators craft challenging, multi-step scenarios based on specific complexity thresholds across dimensions including tool invocation counts, verification conditions, and state dependencies (action ordering is crucial to success), access constraints (e.g. “only team owners can create private channels”) and other policy conflicts (where user request conflicts with system policies). We constrain tasks to have a unique final state, though multiple valid paths may exist to reach it. In addition, we design a subset of 30 infeasible tasks where completion is intentionally impossible due to insufficient tool availability, explicit policy violations, or resource unavailability. For each task, annotators also update the sandbox environment with any additional necessary tables and tools. We follow this with Ground Truth Execution and Plan Authoring, where for each task, annotators provide a detailed step-by-step plan and manually execute it within the sandbox to capture a gold-standard trajectory, documenting each call with parameters, responses and execution rationale. Annotators also author a natural language reasoning plan that explicitly grounds each action in system constraints, user request, and available tools. Plans reference policies from the system prompt, explaining ordering dependencies thus fully grounding the task in the provided context. Refer to Section A.2 for more details.

We enforce Outcome-based Verification, where annotators author executable SQL verification scripts that check the final state of the environment upon task completion. This ensures that we evaluate agents on outcomes rather than rigid action sequences, allowing for alternative valid solution paths. The evaluation includes checks for required conditions (is the goal achieved?), integrity constraints (are foreign key constraints respected?), permission compliance (did the agent avoid unauthorized actions?) and other side effects. Finally, we conduct multiple rounds of Quality Assurance, where reviewer annotators (including authors) assess task feasibility given the initial state and the available tools, instruction clarity and completeness without external dependencies or domain knowledge, verification script correctness and coverage, as well as the fluency and coherence of the ground truth plan, instructions and verification scripts. Refer to Section A.3 for more details on verification.

EnterpriseOps-Gymevaluates agents across 1,150 tasks designed to mimic the depth of real-world enterprise operations, including 30 infeasible tasks that test whether agents correctly refuse unsatisfiable requests. The action space is extensive and diverse, comprising 512 unique tools across domains, with domain-specific toolsets ranging from 37 (Calendar) to 93 (ITSM). Expert human trajectories average 9.15 steps, with planning horizons varying considerably across domains, ranging from 6.2 steps on average in Email to 12.1 in CSM, and reaching up to 34 steps in HR (see Figure 7). Beyond length, our tasks are dense with constraints: on average, a task mandates satisfying 5.3 distinct verification conditions, with the most intricate scenarios requiring the resolution of 44 conditions.

Our sandbox environment models a highly interconnected data ecosystem comprising 164 unique database tables across the eight domains. On average, each task interacts with a sub-graph of 24.9 tables, reaching up to 73 in Hybrid scenarios. This means agents must reason over a large, partially-observable data graph to execute each task correctly. Each task operates over an average of 3,443 database rows, scaling to over 10,000 in data-heavy domains like CSM. To quantify relational complexity, we measure the average number of Foreign Keys (FK) per table. We observe a high degree of dependency, with average FKs ranging from 1.1 in Calendar to 2.4 in HR (mean $\approx$ 1.7), exceeding the relational density of prior benchmarks (see Table 1). Higher FK density means agents must resolve more inter-table dependencies when constructing valid tool arguments, making referential integrity a key challenge. We provide more details in Section A.4.

Original Table 2 — Table 2: Overall task completion performance on EnterpriseOps-Gym. We report the percentage of tasks successfully completed by each model in oracle tool mode, broken down by domain. A task is considered successful only if all outcome verification checks pass.

Table 2
Model	Teams	CSM	Email	ITSM	Calendar	HR	Drive	Hybrid	Infeas.	Avg.
Claude Opus 4.5 (Anthropic, 2025c)	50.0	34.2	51.9	23.8	43.2	32.1	49.5	30.7	50.0	37.4
Gemini-3-Flash (Gemini Team, 2025)	47.3	35.0	44.3	28.5	30.5	12.6	49.7	24.2	38.5	31.9
GPT-5.2 (High) (OpenAI, 2025)	31.0	34.8	51.0	21.7	38.5	25.0	40.0	22.2	50.0	31.8
Claude Sonnet 4.5 (Anthropic, 2025c)	51.0	16.7	51.3	17.6	34.6	21.6	52.1	28.1	46.2	30.9
GPT-5 (OpenAI, 2025)	26.3	36.4	49.0	18.9	41.3	17.9	34.0	23.5	50.5	29.8
Gemini-3-Pro (Gemini Team, 2025)	43.0	27.7	33.6	22.2	28.8	12.5	46.7	22.9	50.0	28.0
GPT-5.2 (Low) (OpenAI, 2025)	25.0	21.2	43.3	6.7	28.9	13.0	26.7	20.9	53.9	21.9
GPT-5-Mini (OpenAI, 2025)	25.7	15.8	47.4	8.9	28.8	10.7	23.8	22.5	47.4	21.3
Gemini-2.5-Pro (Gemini Team, 2025)	39.3	11.6	31.1	13.9	12.5	4.9	27.0	19.6	34.7	18.2
Open Source Models
DeepSeek-V3.2 (High) (DeepSeek-AI, 2024)	37.0	14.1	47.1	16.1	21.2	16.3	35.2	22.9	53.8	24.5
GPT-OSS-120B (High) (OpenAI et al., 2025)	32.0	16.3	42.3	6.1	35.6	16.3	41.0	19.6	50.0	23.7
DeepSeek-V3.2 (Medium) (DeepSeek-AI, 2024)	35.7	15.4	45.8	9.6	21.5	15.0	27.6	22.9	40.0	22.3
Kimi-K2-Thinking (K-Team, 2025)	30.0	7.1	51.0	12.2	15.4	8.2	39.6	15.7	30.5	19.5
Qwen3-30B (Think) (Yang et al., 2025)	22.0	5.4	51.9	6.7	18.3	7.6	25.7	15.7	36.8	16.8
Qwen3-235B (Inst.) (Yang et al., 2025)	28.0	4.7	38.1	9.3	15.7	7.8	23.8	17.7	30.5	16.1
Qwen3-4B (Think) (Yang et al., 2025)	24.0	3.8	38.4	5.6	5.8	7.1	21.9	15.8	31.6	13.7

We evaluate a diverse set of baselines covering closed-source frontier models, open-source reasoning and non reasoning models. All agents are evaluated under a unified interface with identical task instructions, tool definitions, sandbox environments, and evaluation protocols. Unless stated otherwise, agents operate in an oracle-tool setting where we assume a perfect retriever supplies the agent with the right set of tools. This focuses the evaluation purely on planning and execution, without the need for explicit tool discovery. Additionally, we conduct ablations by increasing the number of available tools to analyze how tool set size impacts performance. We use a standard ReAct-style reasoning and tool-execution loop which has been shown to be effective in agentic settings (Yao et al., 2022). The closed-source set includes Claude 4.5 (Anthropic, 2025c; Anthropic, 2025a) variants (Opus and Sonnet), GPT (OpenAI, 2025) variants (5.2 High, 5.2 Low, 5, and 5-Mini), and Gemini (Gemini Team, 2025) variants (3-Pro, 3-Flash, and 2.5-Pro), while the open-source set includes Kimi-K2-Thinking (K-Team, 2025), DeepSeek-V3.2 (DeepSeek-AI, 2024), GPT-OSS-120B (Medium) (OpenAI et al., 2025), and Qwen3 (Yang et al., 2025) variants (235B Inst., 30B Think, and 4B Think).

We evaluate models using pass@1 task completion rate, where a model receives a score of 1 only when it successfully completes all task requirements while satisfying all specified constraints. Task completion is verified by executing SQL-based verifiers hand-written by subject matter experts (SMEs) during the benchmark curation process. We report the average of pass@1 across three runs (to reduce variance) as our primary metric because it captures end-to-end task success. While we also measure verifier-level success rates (see Table 6), which provide fine-grained insight into the average number of successful verification checks, this metric can be misleading: agents may pass verifiers for trivial trajectory segments (e.g., initial setup steps) while failing on core task logic, system compliance requirements, or side-effect checks. Pass@1 therefore provides a more accurate assessment of real-world agent utility.

Overall, Claude Opus 4.5 achieves the best average task completion (37.4%) and is particularly strong across several workflows, leading on Email (51.9%), Calendar (43.2%), HR (32.1%), and Hybrid (30.7%). Gemini-3-Flash emerges as the second-best model overall (31.9%) and tops ITSM (28.5%), a service and operations management workflow. Claude Sonnet 4.5 (30.9%) remains strong on collaboration and document-centric workflows, leading on Teams (51.0%) and Drive (52.1%). GPT-5 shows more domain-specific peaks, topping CSM (36.4%). Open-source models still lag behind the closed-source systems overall. The strongest open-source model, DeepSeek-V3.2 (High), reaches a 24.5% average, narrowly ahead of GPT-OSS-120B (High) at 23.7%. They particularly struggle on service, policy, and people-facing domains such as CSM, ITSM, and HR. Qwen3-30B (Think) performs surprisingly well for its size, outperforming its larger instruct variant and attaining a highly competitive Email score (51.9%, tied for best). Finally, ITSM and Hybrid cross-domain workflows are the hardest settings (best: 28.5% and 30.7% respectively), highlighting that service operations and cross-domain coordination remain the key bottlenecks for all model families.

We assessed robustness to tool overload by conducting ablations with Claude Sonnet 4.5, chosen for its strong overall performance and cost-effectiveness. We augmented the oracle toolset with extra distractor tools (+5, +10, and +15). To make this setting particularly challenging and representative of realistic retrieval errors, we asked Claude to retrieve the distractor tools that appeared most relevant to the task. Surprisingly, performance remained remarkably stable. The average completion rates actually increased slightly by an average of $\sim$ 1.0% (+0.07% for +5, +2.35% for +10, and +0.64% for +15 tools). The only other notable variation was an average 4–9% increase in output tokens. This suggests that the model utilizes the additional token budget to carefully filter and select the appropriate tools. Such robustness likely stems from the extensive tool-use training inherent in these LLMs. Consequently, our findings indicate that the primary bottleneck for these agents is not tool discovery, but rather task planning and adherence to system policies.

Which model offers the best cost–performance trade-off depends on whether the priority is absolute quality or quality per dollar. As shown by the Pareto frontier in Figure 1, Gemini-3-Flash provides the strongest practical trade-off among closed-source models. It achieves 31.9% performance at 0.03 USD per task, delivering a higher success rate than more expensive models like GPT-5 (29.8% at 0.16 USD) and Claude Sonnet 4.5 (30.9% at 0.26 USD) at a fraction of the cost (80–90% less). Within the open-source ecosystem, DeepSeek-V3.2 (High) emerges as the Pareto-dominant option, achieving 24.5% performance at just 0.014 USD closely followed by GPT-OSS-120B (High) (23.7% at 0.015 USD), making these the best open-source value overall. Qwen3-235B (Inst.) remains the cheapest overall option (0.007 USD) but comes with a significant performance floor of 16.1%. Given that success rates across the board remain below 40%, these systems are not yet reliable enough for autonomous deployment without human oversight. For the highest absolute reliability, Claude Opus 4.5 remains the premier choice (37.4%), though it requires a steep premium of 0.36 USD per task.

We curate 30 infeasible tasks across the 8 domains to evaluate whether models appropriately abstain from unsatisfiable requests. Each task has an average of 10 verification checks to ensure there are no side effects on the system. Tasks are impossible through three primary mechanisms: insufficient tool availability, explicit policy violations (e.g. scheduling conflicts, data access rules etc) and resource unavailability (e.g. inactive users, system in migration mode etc). Moreover, tasks employ compound constraints averaging 3 to 4 per request, necessitating that models evaluate multiple intersecting conditions to identify task feasibility. As seen in Table 2, GPT-5.2 (Low) and DeepSeek-V3.2 (High) perform the best (53.9% and 53.8% respectively) in abstaining from the task while leaving no side effects on the system, followed closely by GPT-5 (50.5%) and a cluster of models including Claude Opus 4.5, Gemini-3-Pro, and GPT-OSS-120B (High) at 50.0%. However, the absolute scores of all the models remain well below safe applicability for production systems.

Figure 4: Performance degrades consistently with planning horizon. Pass@1 accuracy for closed-source (solid) and open-weight (dashed) models across horizon lengths 4–16. Thick lines show the group mean $\pm$ 1 SE. We observe monotonic degradation of performance for both sets, while open model performance falls more sharply with horizon length.

Figure 5: Impact of thinking budget on performance Histograms show the performance numbers with thinking budget with GPT-OSS-120B model (OpenAI et al., 2025) across domains. The results show that the model with low thinking budget performing poorly with performance steadily increasing with thinking budget.

To understand how model capabilities degrade with task complexity, we stratify tasks by their expected horizon length, which is proportional to the number of tool execution steps in the human created execution trajectories. As illustrated in Figure 4, performance across all models exhibits a consistent decay as the task horizon increases, reflecting the cumulative difficulty of maintaining reasoning integrity over multi-step sequences. The closed-source group, led by Claude Opus 4.5, demonstrates greater resilience, maintaining a performance lead even as the group mean drops from approximately 35% at 4 steps to under 20% by step 16. In contrast, the open-source cohort shows a much steeper decline, with models like Kimi K2 and GPT OSS 120B converging toward a success rate near 10% at the maximum horizon. This near-universal trend suggests that while current models can navigate short-to-medium sequences, the rapid accumulation of errors in long-horizon tasks remains a critical barrier to autonomous reliability in production environments.

We evaluated the impact of test-time compute by varying the thinking budget (low, medium, high) for the GPT-OSS-120B model (OpenAI et al., 2025). Increasing the thinking budget yields substantial improvements in task completion across almost all domains (see Figure 5). Operating at a low thinking budget causes the model to struggle severely, achieving near-zero accuracy on complex service and people-facing domains such as CSM (1.1%), ITSM (1.1%), and HR (0.0%). Scaling to a high budget unlocks significant capabilities, driving dramatic absolute gains in Drive ( $8.6\to 41.0$ %), Calendar ( $8.7\to 35.6$ %), and Teams ( $4.0\to 32.0$ %). This strong dependence on test-time compute underscores that EnterpriseOps-Gym tasks require complex reasoning and planning to execute. However, we also observe that performance scaling is not universally monotonic; for instance, performance on Email peaks at the medium budget (45.2%) before receding slightly, and ITSM plateaus early ( $1.1\to 6.1\%\to 6.1\%$ ). This suggests that simply allocating more thinking tokens cannot universally overcome fundamental capability bottlenecks in certain workflows.

Original Table 3 — Table 3: Plan-Conditioned Execution Baseline. Comparison of performance with Claude Plans (CP) vs. Human Plans (HP). Green values indicate % improvement over the ReAct baseline from Table 2.

Table 3
Model	Plan	CSM	ITSM	HR
Kimi-K2	CP	$19.6$	$\uparrow$ 12.5%	$18.1$	$\uparrow$ 5.9%	$17.2$	$\uparrow$ 9.0%
HP	$42.2$	$\uparrow$ 35.1%	$29.1$	$\uparrow$ 16.9%	$34.5$	$\uparrow$ 26.3%
Qwen3-30B	CP	$15.2$	$\uparrow$ 9.8%	$11.7$	$\uparrow$ 5.0%	$17.9$	$\uparrow$ 10.3%
HP	$33.9$	$\uparrow$ 28.5%	$20.9$	$\uparrow$ 14.2%	$33.2$	$\uparrow$ 25.6%
Qwen3-4B	CP	$16.8$	$\uparrow$ 13.0%	$12.2$	$\uparrow$ 6.6%	$19.6$	$\uparrow$ 12.5%
HP	$37.16$	$\uparrow$ 33.4%	$23.3$	$\uparrow$ 17.7%	$36.4$	$\uparrow$ 29.3%

The results above show that performance degrades sharply with horizon length and that models struggle most with planning rather than execution. To understand this further, we run a series of controlled ablations that both decouple planning from execution and build increasingly complex multi-agent systems to test whether distributing cognitive load across specialized agents can recover performance that a single monolithic agent cannot achieve.

Automated planning improves weaker models consistently. We introduce a planner-executor baseline in which a dedicated planner agent, instantiated with Claude Sonnet 4.5, one of our best peforming model, generates a high-level plan reasoning over user intent, policy constraints, and potential side effects. A separate executor then carries out tool execution using the same ReAct loop as the single-agent baseline. We evaluate three weaker models (Kimi-K2, Qwen-30B, and Qwen-4B) on the three most challenging domains (CSM, ITSM, and HR). As shown in Table 3, performance improves consistently across all models and domains with gains of 6-13% confirming that planning quality is a meaningful bottleneck even when the executor model is fixed.

Figure 6: Impact of agentic orchestration on performance Histograms show the performance numbers with various multi-agentic architectures using Claude-Sonnet-4.5. The baseline is the simple ReAct loop described in Section 4.1. Planner+Executor architecture first prompts the model for a detailed plan, and performs a ReAct loop conditioned on the plan. Planner+Decompose+Subtask Executor additionally does a task decomposition and calls subagents in a ReAct loop for each subtask. Finally, Oracle Human Plan + Executor performs a ReAct execution loop conditioned on a human written plan.

Human-authored plans reveal a substantially higher ceiling. To bound what better planning could achieve, we give the same executor models human-authored reference plans and ask them to carry out the corresponding tool execution, fully decoupling planning from execution. The gains are considerably larger than those from automated planning: 14-35 percentage points across models and domains, roughly doubling the improvements seen from Claude-generated plans. Notably, Qwen3-4B with human-authored plans (and Claude plans) is competitive with or outperforms larger models under the same condition. This suggests that when strategic reasoning is externalized, the primary remaining challenge is faithful instruction-following and precise tool invocation, both capabilities in which modern LLMs exhibit broad competence regardless of scale. This may also indicate that larger models, having stronger internal priors, are more prone to deviating from a provided plan, while smaller models tend to follow step-by-step instructions more literally—an advantage when those instructions are optimal. However, gap between the human plans and LLM plans indicates that current LLMs fall well short of human-level strategic reasoning on these tasks.

More complex orchestration does not close this gap. We evaluate two MAS configurations for Claude Sonnet 4.5: a Planner+Executor system that conditions ReAct on an auto-generated plan, and a Planner+Decompose+Subtask Executor system that additionally decomposes tasks and invokes a separate subagent per subtask. We evaluate these systems against the ReAct executor baseline with and without human authored plans. As shown in Figure 6, the Planner+Executor setup consistently outperforms the ReAct baseline, yielding absolute gains of 10.7% in CSM and 8.8% in HR. However, the decomposition architecture is less robust. While it provides a minor lift in ITSM, it regresses in both CSM and HR, even falling below the base ReAct performance in CSM (16.2% vs. 16.7%). This is consistent with EnterpriseOps-Gym tasks having strong sequential state dependencies that decomposition disrupts. Ultimately, the substantial remaining gap between automated systems and ReAct with human plans suggests that progress requires advances in constraint-aware plan generation rather than architectural complexity alone.

We performed a manual qualitative analysis of samples where models made partial progress but ultimately failed to complete the task. We observe several recurring failure patterns. Models frequently invoke tools that create database objects without first querying the necessary prerequisites, producing dangling records with broken foreign-key links (Missing Prerequisite Lookup). For example, in a task requiring the creation of an HR topic under a specific category, the model skips retrieving available categories and inserts an orphaned record. Models also fail to trigger the follow-up actions mandated by system policies when certain state transitions occur (Cascading State Propagation). Further failure modes include passing unverified identifiers to tool calls instead of resolving the correct IDs through prior tool interactions (Incorrect ID Resolution), and prematurely declaring task completion before all required steps have been executed (Premature Completion Hallucination). To futher systematically categorize these errors, we use an LLM (Claude Sonnet 4.5 (Anthropic, 2025b)) to tag the final-state SQL verifiers into three types based on their expert-written descriptions: i) Task Completion verifiers check whether the primary user objective was achieved; ii) Integrity Constraints verifiers check that the system remains in a consistent state with valid foreign-key relationships; and iii) Permission and Process Compliance verifiers check adherence to system policies governing permissions and procedural rules. Verifier pass rates for these categories are reported in Table 7. Models struggle most with Permission and Process Compliance—a particularly critical gap for real-world deployment, where policy violations can cause cascading system failures and introduce serious security vulnerabilities. Refer to Appendix C for examples of model failures.

We introduced EnterpriseOps-Gym, a benchmark and sandboxed evaluation platform spanning 1,150 expert-curated tasks across eight enterprise productivity domains, with 512 tools and SQL-based verifiers authored by subject matter experts. Our experiments surface several findings that we believe have broad implications for the development of enterprise-grade LLM agents.

Current agents are far from enterprise-ready. Even under oracle tool access—the most favorable possible retrieval setting—the best model achieves only 37.4% task success, and performance degrades monotonically with horizon length and cross-domain coupling. Critically, failure modes are not random: our qualitative and quantitative analysis shows that agents struggle most with Permission and Process Compliance (e.g., policy adherence, cascading state transitions) rather than with basic task completion. Furthermore, even frontier models refuse infeasible tasks reliably only about half the time (best: 53.9%), falling well short of the robustness required for unsupervised deployment. These results indicate that the gap to enterprise reliability will not be closed by scaling model capacity alone.

Planning is the dominant bottleneck, not tool execution. Our plan-conditioned ablations demonstrate that human-authored plans yield 14–35 percentage point gains across models and domains which is far larger than gains from automated planning or more complex multi-agent orchestration. Strikingly, Qwen3-4B conditioned on human plans is competitive with much larger models under the same condition, suggesting that once strategic reasoning is externalized, even small models can execute faithfully. This dissociation between planning and execution ability implies that the core challenge is constraint-aware plan generation, not tool invocation proficiency. Consistent with this, distractor tools do not meaningfully hurt performance further confirming that tool retrieval is not the binding constraint. Advances in long-horizon, policy-aware planning are therefore the highest-leverage direction for improving agent performance on EnterpriseOps-Gym.

Thinking budget matters, but has domain-specific ceilings. Increasing test-time compute yields substantial gains in most domains, but scaling is not universally monotonic: some domains plateau early, suggesting that additional reasoning tokens cannot compensate for fundamental gaps in domain knowledge or policy understanding. Future work should investigate how to allocate test-time compute more adaptively, and whether targeted training on constraint-heavy domains can raise these ceilings.

Future directions. Our results motivate three concrete research priorities. First, constraint-aware plan generation: methods that explicitly reason over policy constraints, side-effect dependencies, and prerequisite structures before committing to action sequences. Second, long-horizon state management: mechanisms for maintaining coherent world state over many tool calls, such as episodic memory or structured state representations, to prevent the error accumulation we observe with increasing horizon length. Third, safe refusal and escalation: agents must reliably detect infeasible or policy-violating requests and abstain cleanly, a capability that remains weak across all evaluated systems today.

We will release EnterpriseOps-Gym, its sandbox environment, and evaluation tooling to support open, community-driven research. The sandbox is modular and extensible, allowing new domains, tools, and workflows to be added as enterprise practices evolve. By grounding agent evaluation in realistic, constraint-rich enterprise workflows, EnterpriseOps-Gym aims to shift the field’s focus toward the planning, safety, and policy-compliance capabilities that truly determine whether an LLM agent is deployable as a reliable AI worker.

We gratefully acknowledge Turing as our data curation partner for this project. We extend special thanks to Ankit Jasuja, Aakash Chavan, Harshil Parekh, Anuj Jain, Igor Vidal, Rahul Bora, and Sudarshan Sivaraman from Turing, whose instrumental contributions to sandbox engineering and sample generation upheld the highest standards of quality throughout. We also thank the more than 160 data contributors and engineers who painstakingly authored benchmark samples under specific guidance and iterative feedback from the authors — their dedication was indispensable to the scale and fidelity of this benchmark. This work further benefited from the meticulous quality assurance oversight provided by the linguistics team at ServiceNow, particularly Racheal Hansen, Tiffany Do, and Nidhi Kumari. We also thank Rabiul Awal for his helpful feedback on an early draft of this paper. Finally, we gratefully acknowledge Patrice Bechard and Vikas Yadav at ServiceNow for their valuable feedback throughout the development of this work.

References

Anthropic (2025a) Anthropic Introducing claude opus 4.5. External Links: Link Cited by: §4.1.Source
Anthropic (2025b) Anthropic Introducing claude opus 4.5. Cited by: §4.4.
Anthropic (2025c) Anthropic Introducing claude sonnet 4.5. External Links: Link Cited by: Appendix C, Table 6, Table 6, §1, §4.1, Table 2, Table 2.Source
Barres et al. (2025) V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan $\tau^{2}$-Bench: evaluating conversational agents in a dual-control environment. External Links: 2506.07982, Link Cited by: §2, Table 1.Source
Boisvert et al. (2024) L. Boisvert, M. Thakkar, M. Gasse, M. Caccia, T. L. S. D. Chezelles, Q. Cappart, N. Chapados, A. Lacoste, and A. Drouin WorkArena++: towards compositional planning and reasoning-based common knowledge work tasks. External Links: 2407.05291, Link Cited by: §1, §2, Table 1.Source
Bonatti et al. (2025) R. Bonatti, D. Zhao, F. Bonacci, D. Dupont, S. Abdali, Y. Li, Y. Lu, J. Wagle, K. Koishida, A. Bucker, L. K. Jang, and Z. Hui Windows agent arena: evaluating multi-modal OS agents at scale. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: §2.Source
Chen et al. (2025) C. Chen, X. Hao, W. Liu, X. Huang, X. Zeng, S. Yu, D. Li, S. Wang, W. Gan, Y. Huang, W. Liu, X. Wang, D. Lian, B. Yin, Y. Wang, and W. Liu ACEBench: who wins the match point in tool usage?. External Links: 2501.12851, Link Cited by: §1, §2, Table 1.Source
DeepSeek-AI (2024) DeepSeek-AI DeepSeek-v3 technical report. External Links: 2412.19437, Link Cited by: Table 6, Table 6, §4.1, Table 2, Table 2.Source
Drouin et al. (2024) A. Drouin, M. Gasse, M. Caccia, I. H. Laradji, M. D. Verme, T. Marty, L. Boisvert, M. Thakkar, Q. Cappart, D. Vazquez, N. Chapados, and A. Lacoste WorkArena: how capable are web agents at solving common knowledge work tasks?. External Links: 2403.07718 Cited by: §1, §1, §2, Table 1.
Froger et al. (2025) R. Froger, P. Andrews, M. Bettini, A. Budhiraja, R. S. Cabral, V. Do, E. Garreau, J. Gaya, H. Laurençon, M. Lecanu, K. Malkan, D. Mekala, P. Ménard, G. M. Bertran, U. Piterbarg, M. Plekhanov, M. Rita, A. Rusakov, V. Vorotilov, M. Wang, I. Yu, A. Benhalloum, G. Mialon, and T. Scialom ARE: scaling up agent environments and evaluations. External Links: 2509.17158, Link Cited by: §2.Source
Gemini Team (2025) Gemini Team Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. External Links: 2507.06261, Link Cited by: Table 6, Table 6, Table 6, §4.1, Table 2, Table 2, Table 2.Source
He et al. (2025) W. He, Y. Sun, H. Hao, X. Hao, Z. Xia, Q. Gu, C. Han, D. Zhao, H. Su, K. Zhang, M. Gao, X. Su, X. Cai, X. Cai, Y. Yang, and Y. Zhao VitaBench: benchmarking llm agents with versatile interactive tasks in real-world applications. External Links: 2509.26490, Link Cited by: §2.Source
Huang et al. (2025a) K. Huang, A. Prabhakar, S. Dhawan, Y. Mao, H. Wang, S. Savarese, C. Xiong, P. Laban, and C. Wu CRMArena: understanding the capacity of llm agents to perform professional crm tasks in realistic environments. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Cited by: §1, §1, §2, Table 1.
Huang et al. (2025b) K. Huang, A. Prabhakar, O. Thorat, D. Agarwal, P. K. Choubey, Y. Mao, S. Savarese, C. Xiong, and C. Wu CRMArena-pro: holistic assessment of llm agents across diverse business scenarios and interactions. arXiv preprint arXiv:2505.18878. Cited by: §1, §1, §2, Table 1.
Jha et al. (2025) S. Jha, R. Arora, Y. Watanabe, T. Yanagawa, Y. Chen, J. Clark, B. Bhavya, M. Verma, H. Kumar, H. Kitahara, N. Zheutlin, S. Takano, D. Pathak, F. George, X. Wu, B. O. Turkkan, G. Vanloo, M. Nidd, T. Dai, O. Chatterjee, P. Gupta, S. Samanta, P. Aggarwal, R. Lee, P. Murali, J. Ahn, D. Kar, A. Rahane, C. Fonseca, A. Paradkar, Y. Deng, P. Moogi, P. Mohapatra, N. Abe, C. Narayanaswami, T. Xu, L. R. Varshney, R. Mahindru, A. Sailer, L. Shwartz, D. Sow, N. C. M. Fuller, and R. Puri ITBench: evaluating ai agents across diverse real-world it automation tasks. External Links: 2502.05352, Link Cited by: §1, §2, Table 1.Source
Jimenez et al. (2024) C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan SWE-bench: can language models resolve real-world github issues?. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §1.Source
K-Team (2025) K-Team Kimi k2: open agentic intelligence. External Links: 2507.20534, Link Cited by: Table 6, §4.1, Table 2.Source
Li et al. (2023) M. Li, Y. Zhao, B. Yu, F. Song, H. Li, H. Yu, Z. Li, F. Huang, and Y. Li API-bank: a comprehensive benchmark for tool-augmented LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, pp. 3102–3116. External Links: Link, Document Cited by: §1, §2, Table 1.Source Source
Nayak et al. (2025) S. Nayak, X. Jian, K. Q. Lin, J. A. Rodriguez, M. Kalsi, R. Awal, N. Chapados, M. T. Özsu, A. Agrawal, D. Vazquez, C. Pal, P. Taslakian, S. Gella, and S. Rajeswar UI-vision: a desktop-centric gui benchmark for visual perception and interaction. External Links: 2503.15661, Link Cited by: §2.Source
OpenAI et al. (2025) OpenAI, :, S. Agarwal, L. Ahmad, J. Ai, S. Altman, A. Applebaum, E. Arbus, R. K. Arora, Y. Bai, B. Baker, H. Bao, B. Barak, A. Bennett, T. Bertao, N. Brett, E. Brevdo, G. Brockman, S. Bubeck, C. Chang, K. Chen, M. Chen, E. Cheung, A. Clark, D. Cook, M. Dukhan, C. Dvorak, K. Fives, V. Fomenko, T. Garipov, K. Georgiev, M. Glaese, T. Gogineni, A. Goucher, L. Gross, K. G. Guzman, J. Hallman, J. Hehir, J. Heidecke, A. Helyar, H. Hu, R. Huet, J. Huh, S. Jain, Z. Johnson, C. Koch, I. Kofman, D. Kundel, J. Kwon, V. Kyrylov, E. Y. Le, G. Leclerc, J. P. Lennon, S. Lessans, M. Lezcano-Casado, Y. Li, Z. Li, J. Lin, J. Liss, Lily, Liu, J. Liu, K. Lu, C. Lu, Z. Martinovic, L. McCallum, J. McGrath, S. McKinney, A. McLaughlin, S. Mei, S. Mostovoy, T. Mu, G. Myles, A. Neitz, A. Nichol, J. Pachocki, A. Paino, D. Palmie, A. Pantuliano, G. Parascandolo, J. Park, L. Pathak, C. Paz, L. Peran, D. Pimenov, M. Pokrass, E. Proehl, H. Qiu, G. Raila, F. Raso, H. Ren, K. Richardson, D. Robinson, B. Rotsted, H. Salman, S. Sanjeev, M. Schwarzer, D. Sculley, H. Sikchi, K. Simon, K. Singhal, Y. Song, D. Stuckey, Z. Sun, P. Tillet, S. Toizer, F. Tsimpourlas, N. Vyas, E. Wallace, X. Wang, M. Wang, O. Watkins, K. Weil, A. Wendling, K. Whinnery, C. Whitney, H. Wong, L. Yang, Y. Yang, M. Yasunaga, K. Ying, W. Zaremba, W. Zhan, C. Zhang, B. Zhang, E. Zhang, and S. Zhao Gpt-oss-120b & gpt-oss-20b model card. External Links: 2508.10925, Link Cited by: Table 6, Figure 5, Figure 5, §4.1, §4.3, Table 2.Source
OpenAI (2025) OpenAI Introducing gpt-5. External Links: Link Cited by: Table 6, Table 6, Table 6, Table 6, §1, §4.1, Table 2, Table 2, Table 2, Table 2.Source
Qian et al. (2025) C. Qian, Z. Liu, A. Prabhakar, Z. Liu, J. Zhang, H. Chen, H. Ji, W. Yao, S. Heinecke, S. Savarese, C. Xiong, and H. Wang UserBench: an interactive gym environment for user-centric agents. External Links: 2507.22034, Link Cited by: §2.Source
Qin et al. (2024) Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, S. Zhao, L. Hong, R. Tian, R. Xie, J. Zhou, M. Gerstein, dahai li, Z. Liu, and M. Sun ToolLLM: facilitating large language models to master 16000+ real-world APIs. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §1, §2.Source
Styles et al. (2024) O. Styles, S. Miller, P. Cerda-Mardini, T. Guha, V. Sanchez, and B. Vidgen WorkBench: a benchmark dataset for agents in a realistic workplace setting. External Links: 2405.00823, Link Cited by: §2, Table 1.Source
Vishwakarma et al. (2025) H. Vishwakarma, A. Agarwal, O. Patil, C. Devaguptapu, and M. Chandran Can llms help you at work? a sandbox for evaluating llm agents in enterprise environments. External Links: 2510.27287, Link Cited by: §1, §2, Table 1.Source
Xie et al. (2024) T. Xie et al. OSWorld: benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972. Cited by: §2.
Xu et al. (2024) F. F. Xu, Y. Song, B. Li, Y. Tang, K. Jain, M. Bao, Z. Z. Wang, X. Zhou, Z. Guo, M. Cao, M. Yang, H. Y. Lu, A. Martin, Z. Su, L. Maben, R. Mehta, W. Chi, L. Jang, Y. Xie, S. Zhou, and G. Neubig TheAgentCompany: benchmarking llm agents on consequential real world tasks. External Links: 2412.14161, Link Cited by: §1, §2, Table 1.Source
Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: Table 6, Table 6, Table 6, §4.1, Table 2, Table 2, Table 2.
Yao et al. (2024) S. Yao, N. Shinn, P. Razavi, and K. Narasimhan $\tau$-Bench: a benchmark for tool-agent-user interaction in real-world domains. External Links: 2406.12045, Link Cited by: §2, Table 1.Source
Yao et al. (2022) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao ReAct: synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629. Cited by: §4.1.
Zhou et al. (2024) S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig WebArena: a realistic web environment for building autonomous agents. External Links: 2307.13854, Link Cited by: §2.Source

Table of Contents

Page

Paper table
A. Data Collection and Human Annotation .	A
A.1 Demographics and Recruitment .	A.1
A.2 Annotation Process .	A.2
A.3 Quality Assurance and Verification .	A.3
A.4 Sandbox Environment .	A.4
B. Task Categories and Examples .	B
B.1 Calendar .	B.1
B.2 Customer Service Management (CSM) .	B.2
B.3 Drive .	B.3
B.4 Email .	B.4
B.5 Human Resources (HR) .	B.5
B.6 IT Service Management (ITSM) .	B.6
B.7 Teams .	B.7
B.8 Hybrid .	B.8
C. Rollout examples .	C
D. Additional Analysis and ResultsRollout examples .	D
E. Impact Statement .	E

We partnered with a professional data annotation vendor (Turing $^{2}$ ) specializing in data curation for AI applications. The annotation team was structured as a multi-tiered workforce consisting of annotators, quality assurance reviewers, and team leads. The contributors were distributed around major geo locations including Asia, North America, Latin America and Africa, with an age range of 22-37 years. All data contributors held bachelor’s degrees in Engineering, Computer Science, or related disciplines and possessed prior experience in data labeling and UI research.

Recruitment selection prioritized strong proficiency in technical writing, English, and computer science fundamentals, along with expertise in prompt engineering. To ensure high domain realism, the team also included Subject Matter Experts (SMEs) for technical domains such as ITSM and CSM, as well as software engineers responsible for building the sandbox environments. On average, each domain was supported by 20 annotators and 6 reviewers, totaling over 160 contributors. The data collection campaign spanned approximately four months, beginning with a one-month pilot phase. During this pilot period, we collaborated closely with the vendor’s team to conduct detailed reviews and provide extensive feedback, enabling contributors to refine their understanding of the task requirements. All contributors were fairly compensated. The creation of each task, including scenario design, verification, and quality assurance, cost approximately 100 USD.

The data generation pipeline began with contributors being assigned to specific domains and taxonomies. They were supported by a simulated environment that included domain-specific databases and a set of available functions. Annotators and reviewers utilized an internally developed tool to streamline the process.

To ensure a diverse range of difficulty, contributors were given specific complexity thresholds based on the number of required tools and verification steps used to complete a task. A higher number of tools and verifiers directly correlated with higher task complexity. Annotators leveraged their domain-specific expertise to design complex scenarios and problems within their assigned taxonomies, while internal tooling captured their action trajectories.

Verification was rigorous, multi-layered, and designed to retain only the most challenging tasks. Upon completion of a trajectory and its verification scripts, we employed a preliminary filtration and verification stage using state-of-the-art LLMs, specifically GPT-5, Gemini, and Claude. We executed draft tasks against these models and analyzed the resulting trajectories to identify failure modes such as incorrect task definitions leading to unintended paths, missing tools, invalid database entries, or access control conflicts. Additionally, this automated stress-testing flagged overly simple reasoning paths and issues with groundedness. These insights enabled annotators to refine task definitions, database states, and tool availability, while simultaneously discarding trivial tasks. This iterative process naturally drove the creation of more well-defined and complex scenarios. Following this automated phase, tasks underwent human verification, where reviewers evaluated each entry for grammatical accuracy, tool usage logic, natural language fluency, and execution correctness. Detailed quality rubrics were employed to standardize assessments of trajectory quality, prompt clarity, and verifier robustness. This rigorous pipeline ensured that only high-quality, complex tasks were retained in the final benchmark.

The seed data for EnterpriseOps-Gym is generated with a strong real-world perspective with the help of domain SMEs. For each domain, we studied publicly available official API documentation, data models, and usage examples from relevant enterprise systems to understand:

•
Typical entity structures and relationships
•
Field semantics and data constraints
•
Expected API behavior and response patterns

Guided by this research and the domain expertise of our SMEs, we modeled realistic behavior and constraints for each table and field. Our primary objective was to ensure structural realism and behavioral fidelity while remaining platform-agnostic, avoiding reliance on any specific vendor’s proprietary dataset. Furthermore, engineers and SMEs conducted rigorous testing to verify database consistency, ensuring the absence of missing elements or logical contradictions.

The seed data varies significantly between database files for every task. While certain individual field values (e.g., names or email patterns) may occasionally repeat, each dataset represents a distinct high-level use case. The overall data composition, relationships, and scenarios are intentionally unique rather than simple surgical variations of the same seed. Furthermore, as annotators vetted and designed new tasks, the databases were dynamically expanded with additional tables and entries to meet evolving scenario requirements. This iterative enrichment yielded a highly complex data ecosystem comprising 164 unique tables with a dense interconnectivity (mean foreign key degree of 1.7), ensuring a rich and realistic state space for agentic planning.

We provide the complete evaluation environment as a fully containerized Docker setup, hosting both the domain-specific database infrastructure and the tool execution layer. To ensure reproducibility and isolation, a fresh database instance is initialized for each task run, preventing state leakage or side effects from prior executions. The containerized architecture abstracts the complexity of tool invocation and response handling, providing a consistent interface for agents. This design also simplifies verification, as the environment state can be deterministically queried. Given the rich schema of 164 interconnected tables, the environment is highly extensible; researchers can straightforwardly define new tasks that reflect complex enterprise use cases. We will release these containers, along with a comprehensive guide on adding new tasks and verification scripts, to facilitate further research.

This section details the eight distinct domains within the benchmark, outlining their unique operational environments, some required reasoning skills, and task examples.

The Calendar category represents a complex administrative environment where the agent manages time, access, and resources with high precision. It goes beyond simple meeting bookings, requiring the agent to act as a system administrator who restructures entire calendar ecosystems and enforces compliance policies. This involves granular management of access control lists for users and groups, setting up programmatic listeners for security audits, and handling metadata for resources like conference rooms.

The reasoning required for these tasks is often sequential and conditional, where the agent must evaluate complex predicates before acting. For instance, it might need to check if a calendar exists or verify an attendee’s status before making changes. This demands strong temporal reasoning to handle timezone conversions and recurrence rules, as well as the ability to enforce policies by translating high-level intents into low-level actions. Agents must handle instructions like “If the event duration is 60 minutes, set color to Green; otherwise, set to Red” or fuzzy requests such as finding “events containing ’sprint’ in the title.”

This category simulates the high-stakes workflow of a B2B technical support center, where the agent acts as a Technical Support Operations Specialist. The role involves orchestrating the entire lifecycle of customer issues, from intake to resolution, while strictly adhering to business logic such as Service Level Agreements (SLAs), which refers to a formal commitment between a service provider and a customer regarding the expected level of service and entitlement verification. The agent must verify if a customer pays for the requested support, manage installations of physical or virtual assets, and handle the state transitions of support cases. Tasks range from ‘‘Register these 5 new servers with these serials… and add the ’Gold Support’ package” to handling SLA breaches where an agent must “Escalate to ’Critical’, assign to the Escalation Manager, and draft an apology email.”

Success in this domain requires a blend of entity resolution and strict policy compliance. Agents must identify the correct assets from vague descriptions (e.g., “The server ending in 9300”) and strictly follow business rules, such as prioritizing VIP accounts regardless of minor issue severity. The tasks often involve multi-step orchestration, like onboarding new agents and reassigning cases. For example, “Transfer her high-priority cases to Bob, her low-priority ones to the Queue, and deactivate her profile”. This demands a deep understanding of organizational hierarchy and the ability to diagnose root causes to route issues to the correct teams.

The Drive category places the agent in the role of a Digital Asset Manager or Information Architect, responsible for the structural integrity and security of corporate file systems. Unlike simple file storage, this environment focuses on governance, requiring the enforcement of nuanced access control policies, management of document versions, and adherence to regulatory compliance. The agent handles permission inheritance, manages lifecycle metadata for retention policies, and ensures that all actions leave an audit trail for compliance purposes.

Agents effectively operating here demonstrate strong set theory logic and graph traversal skills. They must perform operations like “Remove all external users EXCEPT partner.com,” and navigate complex folder hierarchies to reorganize content. The work requires constructing precise search queries from natural language intents, such as identifying “large old videos” using filters like ”mimeType contains ’video’ AND size > 100MB”. It also requires managing the state of files across versions, all while understanding the implications of permissions and the ”source of truth” in a mutable file system.

Representing the work of an intelligent Executive Assistant or Mailbox Administrator, the Email category involves complex mailbox orchestration. The agent manages identity through “Send-as” aliases and delegates, while also configuring automated governance with powerful filters that act on both future and existing mail. Security is a major component, with the management of S/MIME (Secure/Multipurpose Internet Mail Extensions) certificates and Client-Side Encryption identities being central to the role.

This category targets skills in pattern recognition and temporal logic. Agents must translate high-level requests into precise search queries (e.g., finding “messages regarding the Q2 budget”) and set up conditional workflows, such as verifying an alias before sending a draft or managing an Out of Office responder. The reasoning is often administrative and social, requiring the agent to distinguish between important communications and noise, and to identify suspicious configurations, such as “Remove the filter that forwards mail to an unknown gmail address.”

The HR category represents a highly sensitive, process-driven domain focused on employee lifecycle management and data privacy. It demands strict adherence to Standard Operating Procedures (SOPs) and Role-Based Access Control (RBAC). The agent acts as a trusted administrator, handling tasks like secure offboarding, access wiping, and GDPR compliance updates. Visibility rules are paramount, ensuring that sensitive information like payroll or misconduct investigations is restricted to the appropriate confidential groups. For example, a “Secure Termination” task requires the agent to “Initiate involuntary separation… trigger legal hold/forensics task, and revoke all physical/digital access immediately.”

confidential groups. For example, a “Secure Termination” task requires the agent to “Initiate involuntary separation… trigger legal hold/forensics task, and revoke all physical/digital access immediately.”

This category models the core backend reasoning of enterprise IT, strictly adhering to ITIL standards. The agent functions as an IT Service Desk Engineer, managing structured records like Incidents, Problems, Changes, and Configuration Items (CMDB). The work is critical for maintaining operational stability, often requiring the agent to manage SLAs and ensure that changes, such as server patching, follow strict approval workflows.

Reasoning in ITSM is relational and causal. Agents must navigate complex entity graphs to link incidents to their root causes and plan remediations, such as in an “Emergency Change Implementation” where the agent must “Log a ’Major Incident’… create an ’Emergency Change’ request to reboot the server… and Resolve the Incident.” They need to calculate priority based on impact and urgency (e.g., finding “High Priority” incidents nearing SLA breach) and translate unstructured user reports into structured database records.

The Teams category encompasses the definition and management of enterprise collaboration spaces. Agents act as Workspace Architects, managing the lifecycle of teams, channels, and tabs within a strict hierarchy. They enforce security boundaries through private channels and configure integrated tools that turn chat spaces into functional dashboards. The domain also involves “Infrastructure as Text,” where structured configuration data is embedded within channel descriptions or messages. For example, “Update the ’Partners’ channel description to include the JSON config: {"partner_tier": "gold"}.”

Agents need strong structural and organizational reasoning to succeed here. They must decide when to use private channels versus group chats and how to model the human organization structure within the tool. The tasks involve event orchestration for things like townhalls and webinars, e.g., “Schedule a ’Q4 All-Hands’ Townhall… Add the CEO as a co-organizer”, as well as the precise management of tags to facilitate targeted communication.

The Hybrid category represents the most complex class of tasks, requiring the agent to usually operate across two of the seven distinct domains simultaneously. This significantly increases the complexity of the environment, with an average of 40 tables compared to the mean of 25 across single-domain tasks. These scenarios simulate realistic enterprise workflows where actions in one system trigger requirements in another, demanding high-level planning and state tracking across disparate APIs.

For example, a task might require the agent to “Check if a product’s warranty extends beyond 2025 in CSM; if so, log a customer interaction and immediately schedule a ’Warranty Discussion’ on the Sales Calendar.” This compels the agent to retrieve information from the CSM database (warranty status), make a decision based on that data, perform a write operation in CSM (logging the interaction), and then context-switch to the Calendar API to schedule an event using details derived from the CSM record. Success depends on maintaining state consistency across both platforms and correctly mapping entities (like Product IDs to Event Descriptions) between them.

To illustrate the nature of benchmark tasks and the types of failures models exhibit, we present abridged Claude-Sonnet-4.5 (Anthropic, 2025c) rollout examples with summary of the user task, system policy and tools, drawn from the ITSM, CSM and HR domains.

We present additional analysis on task complexity in Figure 7, Tables 7 and 6, and full results in Table 2.

Figure 7: Distribution of task complexity across domains in EnterpriseOps-Gym. Histograms show the distribution for number of tool-execution steps required to complete tasks in each domain. Red dashed lines indicate the mean steps per domain. The results highlight substantial variation in interaction depth, with CSM and HR exhibiting longer, heavier-tailed execution traces, while Email and Calendar tasks tend to require shorter, more tightly distributed action sequences. Hybrid and ITSM workflows show moderate to high variance.

Original Table 6 — Table 6: Overall task completion performance on EnterpriseOps-Gym. (Oracle Mode) We report the percentage of tasks successfully completed by each model in oracle tool mode, broken down by domain. A task is considered successful only if all outcome verification checks pass. Results highlight substantial performance degradation on long-horizon and cross-domain (Hybrid) workflows, indicating persistent improvement gap. VL: Verifier Level, O: Overall.

Table 6
Model	Teams	CSM	Email	ITSM	Calendar	HR	Drive	Hybrid	Overall
Claude Opus 4.5 (Anthropic, 2025c)	50.0	84.2	34.2	66.9	51.9	72.6	23.9	54.8	43.3	73.0	32.1	67.0	49.5	75.3	30.7	61.7	37.0	67.7
Gemini-3-Flash (Gemini Team, 2025)	47.3	81.0	35.0	68.1	44.3	70.0	28.5	58.3	30.5	67.0	12.6	54.9	49.7	76.7	24.2	58.3	31.4	65.0
GPT-5.2 (High) (OpenAI, 2025)	31.0	66.3	34.8	67.6	51.0	72.6	21.7	49.9	38.5	71.2	25.0	60.1	40.0	67.9	22.2	54.6	31.3	62.5
Claude Sonnet 4.5 (Anthropic, 2025c)	51.0	81.9	16.7	55.7	51.3	71.7	17.6	49.9	34.6	70.9	21.6	61.4	52.1	76.0	28.1	61.4	34.1	66.1
GPT-5 (OpenAI, 2025)	26.3	59.7	36.4	62.6	49.0	71.2	18.9	38.4	41.3	71.5	17.9	53.9	34.0	62.3	23.5	53.3	30.9	59.1
Gemini-3-Pro (Gemini Team, 2025)	43.0	77.1	27.7	64.6	33.7	63.2	22.2	56.3	28.8	66.1	12.5	53.1	46.7	73.0	22.9	56.6	27.5	62.2
GPT-5.2 (Low) (OpenAI, 2025)	25.0	55.9	21.2	53.6	43.3	68.4	6.7	20.9	28.8	63.4	13.0	37.5	26.7	58.3	20.9	48.1	21.1	47.8
GPT-5-Mini (OpenAI, 2025)	25.7	55.2	15.8	51.6	47.4	71.1	8.9	29.7	28.8	65.3	10.7	42.7	23.8	56.3	22.5	51.4	22.9	52.9
Gemini-2.5-Pro (Gemini Team, 2025)	39.3	66.8	11.6	48.0	31.1	59.8	13.9	44.8	12.5	52.2	4.9	44.9	27.0	57.9	19.6	53.2	20.0	53.5
Open Source Models
DeepSeek-V3.2 (High) (DeepSeek-AI, 2024)	37.0	67.3	14.1	51.7	47.1	67.3	16.1	47.3	21.2	55.8	16.3	50.1	35.2	59.3	22.9	52.1	23.8	54.7
GPT-OSS-120B (High) (OpenAI et al., 2025)	32.0	61.8	16.3	52.8	42.3	64.6	6.1	24.1	35.6	66.8	16.3	49.3	41.0	67.4	19.6	52.4	23.1	52.1
DeepSeek-V3.2 (Medium) (DeepSeek-AI, 2024)	35.7	62.6	15.4	44.4	45.8	66.0	9.6	27.9	21.5	49.0	15.0	48.0	27.6	55.3	22.9	55.9	24.2	51.1
Kimi-K2-Thinking (K-Team, 2025)	30.0	71.2	7.1	40.6	51.0	69.6	12.2	35.5	15.4	54.7	8.2	39.5	39.6	61.4	15.7	49.5	22.4	52.8
Qwen3-235B (Inst.) (Yang et al., 2025)	28.0	58.3	4.7	30.3	38.1	64.0	9.3	35.8	15.7	54.1	7.8	39.1	23.8	55.4	17.7	47.6	18.1	48.1
Qwen3-30B (Think) (Yang et al., 2025)	22.0	52.4	5.4	37.9	51.9	72.0	6.7	35.3	18.3	61.7	7.6	38.5	25.7	51.7	15.7	48.2	19.1	49.7
Qwen3-4B (Think) (Yang et al., 2025)	24.0	53.1	3.8	32.8	38.4	63.1	5.6	32.3	5.8	48.4	7.1	41.4	21.9	55.3	15.8	47.6	15.3	46.8

Original Table 7 — Table 7: Verifier pass rates across domains by category. Models achieve highest scores on Integrity Constraints (system state validity), followed by Task Completion, with Policy Compliance showing the lowest performance. Sample counts vary across categories, with heavier domains such as CSM, ITSM and HR with more emphasis on Integrity Constraints and Policy compliance.

Table 7
Domain	Task Completion	Integrity Constraints	Policy Compliance
Samples	Score	Samples	Score	Samples	Score
Teams	100	82.3	20	90.0	8	81.3
CSM	183	58.3	30	70.0	120	45.5
Email	103	70.9	17	76.5	12	79.2
ITSM	176	53.7	30	50.6	70	30.0
Calendar	104	71.5	8	62.5	19	76.1
HR	182	63.2	47	56.0	94	50.4
Drive	105	75.1	26	87.8	6	66.7
Hybrid	152	60.0	23	72.5	18	61.1

This work contributes a benchmark and evaluation framework that targets a core challenge in modern AI systems: reliable planning and execution over real tools with persistent state. As LLM-based agents are increasingly deployed to automate workflows involving scheduling, communication, document management, and operational support, understanding their failure modes under realistic constraints is critical.

Positive Impacts: EnterpriseOps-Gym provides the research community with a rigorous, reproducible testbed for studying agentic planning, tool selection, and error recovery. By emphasizing outcome-based verification and safety-critical constraints, the benchmark encourages the development of agents that are not only capable, but also reliable and auditable. Progress enabled by this benchmark may lead to safer automation of repetitive and high-friction workflows, reducing human burden and enabling practitioners to focus on higher-level decision-making. More broadly, EnterpriseOps-Gym advances the study of general-purpose tool use and planning, independent of any single enterprise platform.

Potential Risks and Limitations: As with any benchmark that evaluates execution over tool interfaces, there is a risk that systems trained exclusively to optimize benchmark performance may overfit to specific task patterns or verification criteria. Additionally, increased automation of operational workflows may raise concerns around over-reliance on AI systems, particularly if failures are not well understood or monitored. Finally, training and evaluating large models for agentic behavior incurs computational cost, which has environmental implications that should be considered alongside performance gains.

We mitigate these risks by releasing EnterpriseOps-Gym as a diagnostic benchmark, emphasizing analysis over leaderboard ranking, and by ensuring that all environments are synthetic and sandboxed, with no access to proprietary or sensitive data. We hope this work supports responsible research into agentic systems and helps the community build agents that are not only more capable, but also more dependable.

EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings

Abstract

References

Meow summary

Why test agents this way?

The benchmark’s trick

What it reveals

Where the evidence stops

Make notes of your own

Model	Teams	CSM	Email	ITSM	Calendar	HR	Drive	Hybrid	Infeas.	Avg.
Closed Source Models
Claude Opus 4.5 (Anthropic, 2025c)	50.0	34.2	51.9	23.8	43.2	32.1	49.5	30.7	50.0	37.4
Gemini-3-Flash (Gemini Team, 2025)	47.3	35.0	44.3	28.5	30.5	12.6	49.7	24.2	38.5	31.9
GPT-5.2 (High) (OpenAI, 2025)	31.0	34.8	51.0	21.7	38.5	25.0	40.0	22.2	50.0	31.8
Claude Sonnet 4.5 (Anthropic, 2025c)	51.0	16.7	51.3	17.6	34.6	21.6	52.1	28.1	46.2	30.9
GPT-5 (OpenAI, 2025)	26.3	36.4	49.0	18.9	41.3	17.9	34.0	23.5	50.5	29.8
Gemini-3-Pro (Gemini Team, 2025)	43.0	27.7	33.6	22.2	28.8	12.5	46.7	22.9	50.0	28.0
GPT-5.2 (Low) (OpenAI, 2025)	25.0	21.2	43.3	6.7	28.9	13.0	26.7	20.9	53.9	21.9
GPT-5-Mini (OpenAI, 2025)	25.7	15.8	47.4	8.9	28.8	10.7	23.8	22.5	47.4	21.3
Gemini-2.5-Pro (Gemini Team, 2025)	39.3	11.6	31.1	13.9	12.5	4.9	27.0	19.6	34.7	18.2
Open Source Models
DeepSeek-V3.2 (High) (DeepSeek-AI, 2024)	37.0	14.1	47.1	16.1	21.2	16.3	35.2	22.9	53.8	24.5
GPT-OSS-120B (High) (OpenAI et al., 2025)	32.0	16.3	42.3	6.1	35.6	16.3	41.0	19.6	50.0	23.7
DeepSeek-V3.2 (Medium) (DeepSeek-AI, 2024)	35.7	15.4	45.8	9.6	21.5	15.0	27.6	22.9	40.0	22.3
Kimi-K2-Thinking (K-Team, 2025)	30.0	7.1	51.0	12.2	15.4	8.2	39.6	15.7	30.5	19.5
Qwen3-30B (Think) (Yang et al., 2025)	22.0	5.4	51.9	6.7	18.3	7.6	25.7	15.7	36.8	16.8
Qwen3-235B (Inst.) (Yang et al., 2025)	28.0	4.7	38.1	9.3	15.7	7.8	23.8	17.7	30.5	16.1
Qwen3-4B (Think) (Yang et al., 2025)	24.0	3.8	38.4	5.6	5.8	7.1	21.9	15.8	31.6	13.7

Abstract

1 Introduction

2 Related Works

API and Tool-Use Benchmarks

Enterprise Benchmarks

Agentic Planning and Computer-Use

3 EnterpriseOps-Gym

3.1 EnterpriseOps-GymConstruction

Selecting domains.

Sandbox Environment and Data Generation.

Task Construction Pipeline.

3.2 Dataset Statistics

Task Statistics.

Environment Statistics.

4 Experiments

4.1 Baselines

4.2 Evaluation Metrics

4.3 Results

How do models perform across different domains?

How does adding extra tools affect performance?

Which model offers the best cost–performance tradeoff?

How well do models refuse infeasible tasks?

How does performance scale with task horizon?

How does thinking budget affect performance?

4.4 Further Analysis

Failure Modes

5 Discussion and Conclusion

6 Acknowledgments

References

Appendix

Appendix A Data Collection and Human Annotation

A.1 Demographics and Recruitment

A.2 Annotation Process

A.3 Quality Assurance and Verification

A.4 Sandbox Environment

Environment Setup.

Appendix B Task Categories and Complete Examples

B.1 Calendar

B.2 Customer Service Management (CSM)

B.3 Drive

B.4 Email

B.5 Human Resources (HR)

B.6 IT Service Management (ITSM)

B.7 Teams

B.8 Hybrid

Appendix C Rollout examples (concise)

Appendix D Additional Analysis and Results

Appendix E Impact Statement

Meow summary

Why test agents this way?

The benchmark’s trick

What it reveals

Where the evidence stops

Make notes of your own