EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis

Many studies aim to improve LLMs’ ability to solve tasks with tools (Qu et al., 2025; Luo et al., 2025). In this paper, we focus on general tool use across various domain-specific environments (Patil et al., 2025; Yao et al., 2025; Chen et al., 2025), rather than tool-integrated reasoning and web information access centered on Python or search tools (Dong et al., 2025; Li et al., 2025a). Some work have explored the training data and RL strategies from different perspectives (Prabhakar et al., 2025; Liu et al., 2025b; Xu et al., 2025; Zhang et al., 2026; Zhao et al., 2025). However, they mainly focus on synthetic static trajectories and cannot support LLMs’ self-exploration. For trajectory evaluation, they primarily rely on surface matching, checking whether generated tool names and parameters match references, which is neither sufficient to determine whether the task is truly completed nor able to accommodate multiple equivalent solution paths. In contrast, we synthesize executable environments and tasks, along with rule-based evaluation grounded in environments’ state, thereby supporting LLMs’ training across varied scenarios.

Environments provide agents with action feedback and rewards for interaction and policy optimization. We focus on tool-interactive environments, where LLM agents can use tools to query environmental information or change the state of the environment. One line of work (Guo et al., 2024; Guo et al., 2025; Castellani et al., 2025; Li et al., 2025b) leverages LLMs’ reasoning and world knowledge to simulate environments. Although there is no need to build real environments, it is prone to hallucinations and inconsistencies, and lacks transparency and persistent state management. Another line of work (Tang et al., 2024; Ye et al., 2025; Fang et al., 2025; Cai et al., 2025) builds sandbox environments through programming. However, they either only model isolated, stateless functions, or rely on environmental priors (e.g., trajectories, toolsets) and lack automatic assessment, which limits scalability and coverage. Therefore, we propose EnvScaler to enable automatic, scalable environment and scenario synthesis for agent training.

The first step in scaling environments is to collect diverse environment themes. Unlike manual presetting or derivation from API collections (Fang et al., 2025), SkelBuilder mines them from existing text resources. Considering that studies around SFT have gathered a large and diverse set of tasks that may implicitly contain latent environmental contexts, this inspired us to derive themes through reverse inference from the existing tasks.

Given a task set $T_{\text{exist}}=\{t_{1},\dots,t_{n}\}$, an LLM $M$ first performs binary filtering to retain tasks situated within a domain-specific, stateful environment. For each retained task, $M$ infers the corresponding environment description:

where $P^{\text{task}}_{\text{filter}}$ and $P^{\text{env}}_{\text{infer}}$ denote prompts for task filtering and environment inference$^{1}$. The inferred environments are then aggregated and deduplicated by embedding each description and retaining one record from groups of highly similar descriptions, yielding the final diverse, non-redundant set $\{E_{\text{des}}\}=\text{Dedup}(\{E^{\prime}_{\text{des}}\},\text{sim})$.

To transform the environment description into a programmatically modeled environment, we design a three-stage pipeline.

Logic Planning. An LLM enriches the environment description $E_{\text{des}}$, inferring the Env state definition $E_{\text{state}}$, domain rules $E_{\text{rule}}$, and the list of tool operations $\{E_{\text{tool}_{i}}\}$. These elements serve as a structured blueprint, with $E_{\text{rule}}$ concatenated with $E_{\text{des}}$ to form the environment documentation $E_{\text{doc}}$:

Program Modeling. The LLM first converts the planned state space into class attribute definitions $F_{\text{attr}}$. Then, for each tool operation, given the environment rules and class attributes, it generates the corresponding class-method implementation $F_{\text{meth}_{i}}$, ensuring consistency with rules and proper state transitions:

Program Assembly. The generated code fragments are automatically merged into a complete Python class file $F_{\text{exec}}$, implementing all sandbox logic, where attributes represent environment states and methods represent supported tool operations:

Syntax validity is verified via the abstract syntax tree (AST), with invalid files discarded. AST combined with regex extraction yields all method signatures, forming the tool interface set $\Sigma_{\text{tool}}$ $^{2}$.

Unlike static evaluation via direct LLM scoring, we propose a dual-agent loop to assess the actual tool-execution performance of the environment.

Frontend Testing Agent. After instantiating the environment class, the testing LLM agent $M_{\text{test}}$ has no access to its internal implementation. In round $j$, it receives the current environment state $S_{j}$, and randomly generates a call request, which may be a positive or a negative test case (e.g., invoking a file-deletion tool to delete a non-existent file):

Backend Checking Agent. The environment first executes the tool invocation. The checking agent $M_{\text{check}}$, inspects the tool’s source code $F_{\text{meth}}$, the returned result $R_{j}$, and state changes before and after execution to judge whether the behavior matches expectations:

The testing and checking agents form a closed loop, iterating for $N$ rounds to cover diverse situations. The average judging pass rate serves as the quantitative metric for the environment’s quality:

where environments with $\text{score}_{\text{env}}$ below a predefined threshold are discarded.

In practice, we select API-Bank (Li et al., 2023) and ToolACE (Liu et al., 2025b) as initial task sources, which have high task retention rates. GPT-4.1 and Qwen3-235B-Instruct-2507 are used for environment discovery and programming, while GPT-4.1-mini and Qwen3-30B-A3B-Instruct-2507 are used for environment assessment. The number of test rounds $N$ is set to 100, with a filtering threshold of 0.85. In total, we obtain 191 environments. Table 2 and Figure 10 presents their statistics, with an average of 18.58 tools and 21.38 state categories per environment, reflecting the complexity of the synthesized environments.

In stateful environments, tasks are closely tied to state data. For example, an agent cannot cancel an order that does not exist in the environment’s database. To ensure that the generated task is both solvable and consistent with the given environment and scenario, ScenGenerator first uses an LLM to generate the environment’s initial state data $S_{\text{init}}$, and then derives challenging tasks based on the environment’s initial state:

A common task synthesis approach (Fang et al., 2025; Cai et al., 2025) is to walk through tool invocation sequences and infer the corresponding task by reversing the sequence. However, this often yields low-quality tasks, and the existence of multiple valid task solution paths also make the sequence unsuitable as a unique “ground-truth” reference. Therefore, we directly prompt the LLM to generate a challenging, scenario-specific task from the initial state $S_{0}$, tool set $E_{\text{tool}}$, and rules $E_{\text{rule}}$:

To achieve rule-based trajectory verification, we first use an LLM to decompose the task into a checklist of verifiable conditions. Then, for each checkpoint, we call the LLM to generate a terminal-state validation function. This function takes the environment’s final state $S_{\text{final}}$ after the trajectory as input and returns “True” or “False” to indicate whether the condition is satisfied. Finally, the proportion of passed functions is used as the trajectory’s reward score. Formally, we have:

Compared with a single boolean judgment, decomposing the task into multiple validation functions not only captures partial completion, but also is easier for the LLM to generate. Moreover, unlike superficial evaluation that merely checks whether the tool invocation sequence matches a reference sequence, evaluation based on the environment’s final state is agnostic to the solution process and can accommodate multiple valid solution paths.

The agent–environment interaction can be modeled as a Partially Observable Markov Decision Process (POMDP). At time step $t$, the agent $\pi_{\theta}$ cannot directly access the environment state data $S_{t}$; instead, it makes decisions based on the observation–action history $H_{t}$ and the current observation $o_{t}$:

We consider two interaction settings:$^{3}$

(1) Non-Conversation (Non-Conv.): The environment directly provides the complete task information to the agent, with the initial observation $o_{0}=\{E_{\text{doc}},\Sigma_{\text{tool}},\text{task}\}$. At each step, the agent invokes a tool, and the execution result as the observation, until the agent believes the task is completed or the maximum step limit is reached.

(2) Conversation (Conv.): The environment additionally includes an LLM-simulated user $\pi_{\text{user}}$. Complete task information is not revealed upfront but must be progressively acquired via agent-user dialogue. Compared with Non-Conv, it expands the agent’s action space to include user interaction. The trajectory ends when the user considers the task completed or the maximum steps are reached.

For SFT, trajectories from the teacher LLM can be directly used as learning targets for the student LLM. For RL, trajectories are converted into rewards using the validation functions, which are then applied for policy optimization.

With ScenGenerator, we construct around 7K task scenarios for 191 environments and randomly sample 50 scenarios for a pilot study. As shown in Table 3, Qwen3-8B (Thinking) scores 57.78, while Qwen3-4B (Non-Think) scores only 37.53. Besides, the average trajectory is about 15 steps under the Non-Conv setting, increasing to over 25 steps under Conv. This shows that the synthetic tasks have substantial difficulty and can incentivize LLMs to produce long action trajectories. Moreover, for the same model, multiple sampling and selecting the highest-scoring trajectory significantly boost the score, suggesting that synthetic tasks offer considerable room for self-exploration and optimization. Figure 5 shows that stronger LLMs consistently achieve higher win rates, confirming the validity of the state-check functions in distinguishing and quantifying model performance.

Training. We conduct SFT and RL on Qwen3 series models (Thinking Mode) (Yang et al., 2025). A total of 140 environments are used for SFT, and the remaining 51 environments are used for RL. For SFT, we employ Qwen3-30B-A3B-Thinking-2507 as the teacher model, interacting with the environments under two interaction settings, yielding about 9K trajectories in total. For RL, we apply the Reinforce++ algorithm (Hu et al., 2025) under Non-Conv Setting. Detailed implementation are provided in Appendix B.2.
Evaluation. We use three widely used multi-turn tool-use benchmarks: BFCL-v3 Multi-Turn (abbreviated as BFCL-MT) (Patil et al., 2025), Tau-Bench (Yao et al., 2025), and ACEBench-Agent (Chen et al., 2025). Each benchmark consists of several domain-specific environments equipped with tools, requiring the LLM to interact with users and invoke tools to solve tasks. Notably, ACEBench uses the [func_name(param)] prompt format by default; we modify the official code to support LLMs’ native function-calling interface to ensure consistency. Detailed introductions are provided in Appendix B.1 and Table 9.

As shown in Table 4, using EnvScaler for SFT leads to significant improvements across all benchmarks, and incorporating RL further boosts performance $^{4}$. Next, we analyze the results from three aspects:

(1) Training Strategy. SFT significantly boosts LLMs’ performance. Averaged across three models, BFCL-MT improves by 8.67 points, Tau-Bench by 4.29 points, and ACEBench-Agent by 11.57 points. This can be attributed to the diverse, multi-tool environments and complex tasks synthesized by EnvScaler, which help LLMs strengthen domain adaptability, multi-turn interaction, and multi-tool usage in supervised training. Incorporating RL yields further improvements (the RL curve is shown in Figure 6). For example, Qwen3-8B achieves 4.88 and 3.46 point gains on BFCL-MT and Tau-Bench, respectively. This demonstrates that leveraging synthetic environments and state-checking reward signals can guide LLMs to autonomously learn more effective strategies.

(2) Model Size. SFT delivers stable improvements across models, whereas RL performance is more model-dependent. Qwen3-8B achieves gains across all datasets, whereas Qwen3-1.7B shows notable improvements on BFCL-MT and ACEBench-Agent but a slight drop on Tau-Bench. The main reason is that large-scale models possess stronger exploration capabilities during RL, enabling them to extract effective strategies. In contrast, small-scale models, with weaker foundational abilities, are more susceptible to noisy reward signals and produce lower-quality strategies that struggle to generalize to unseen environments.

(3) Different Benchmarks. Comparatively, EnvScaler shows more substantial gains on BFCL-MT and ACEBench-Agent, while improvements on Tau-Bench are relatively limited. Qwen3-1.7B even slightly declines on the most challenging Tau-Bench Airline task. This is mainly because BFCL-MT and ACEBench-Agent underscore the evaluation of LLM’s ability in multi-turn, multi-tool collaboration and domain adaptability, while EnvScaler can substantially enhance these capabilities. In contrast, Tau-Bench focuses on deep reasoning under complex environment’s rules, making such tasks harder for LLMs to learn and generalize.

To investigate how the similarity between synthetic and test environments affects test performance, we compute the similarity between each synthetic environment and those in BFCL-MT, based on textual embeddings of the environment topic and toolset descriptions. According to the similarity scores, environments are divided into three groups: (1) the 50% most similar; (2) the 50% least similar; and (3) a random 50% selection. As shown in Table 5, training on either the most similar or least similar subsets yields only minor differences in overall metrics, and both significantly outperform the baseline. This suggests that the performance gains do not primarily depend on direct similarity between training and test environments, but on the problem-solving patterns and transferable tool-use skills learned from different environments.

To analyze the effect of scaling the number of training environments on model performance, we sample subsets with varying environment counts from the full SFT dataset and conduct SFT on each subset. As shown in Figure 7, Qwen3-4B’s scores on both the BFCL-MT and ACEBench-Agent exhibit a steady upward trend as the number of training environments increases. The most pronounced improvement occurs when scaling from 0 to 20 environments; although the rate of gain slows beyond this point, performance continues to rise overall. These results indicate that scaling up the number and diversity of training environments effectively enhances the model’s adaptability across contexts and improves its task-solving performance.

To investigate the impact of interaction patterns on model training, we conduct SFT under three data settings: Non-Conv, Conv, and their combination (Full). Table 6 reports the BFCL-MT results for Qwen3-8B. The Base and Long-Context subsets have complete task information, whereas the Miss-Func and Miss-Parm subsets lack necessary tools or parameters, requiring the agent to interact with the user to obtain them. It can be observed that SFT (Non-Conv) performs better on Base and Long-Context, while SFT (Conv) achieves higher scores on Miss-Parm. Performance on Miss-Func does not improve, primarily because the training data lacks samples with missing tool types. The SFT (Full) setting yields the highest overall score, highlighting the need to learn both interaction patterns for adaptability in scenarios with either sufficient or incomplete information.

Table 7 shows the performance of models trained with RL directly on synthetic environments without SFT cold start. Figure 8 presents the curves during the RL process. Overall, all models achieve varying degrees of improvement on three benchmarks, indicating that even without SFT initialization, direct RL can still drive self-optimization. However, the gains from direct RL strongly depend on model size and exploration capability. Qwen3-1.7B shows only minor improvement, while Qwen3-8B achieves the most notable gains. This suggests that larger models can leverage the environment more effectively for exploration and learning high-quality policies, whereas smaller models, limited in base ability and exploration, are more vulnerable to sparse or noisy rewards. Besides, compared with “SFT + RL” in Table 4, the overall gains from direct RL remain limited, highlighting that SFT initialization is still important for better policy quality. Combining SFT with RL can more fully exploit the training value of synthetic environments.

To further evaluate the quality of environments, we decompose the quality evaluation into three dimensions and find that Claude-4.5-Sonnet’s scores align highly with manual judgments: (1) Tool-Program Alignment: it evaluates whether the semantic definition of the tool is consistent with its program implementation. (2) Functional Correctness: it evaluates whether the tool behaviors in the environment meet expectations. (3) Code Robustness: it evaluates whether the environment program handles abnormal inputs and boundary conditions reasonably. As shown in Figure 9, the 191 synthesized environments achieve high average scores of 8.5+ across all three dimensions. This proves the overall reliability of the EnvScaler evaluation mechanism.

We evaluate the average token usage and cost of synthesizing a single environment and a single scenario on a small batch. The results are shown in Table 8. Under the GPT-4.1/mini combination, the average cost per environment is about $1 and the cost per scenario is about $0.06. We consider this acceptable. During environment construction, the 100-round dual-agent loop causes high token consumption. If the number of loop rounds is reduced, the cost can be further lowered.

Figures 20 and 21 show the prompts for generating the environment’s initial state and tasks, respectively. Figures 22 and 23 show the prompts for decomposing tasks into checklists and converting each check item into a verification function.

Figures 24 and 25 show the system prompts guiding LLMs complete tasks under the Non-conversation and Conversation settings. Figure 26 shows the system prompt for guiding a LLM to act as a user.

Figure 10 shows detailed statistics for the 191 synthesized environments. By embedding each environment’s theme description and applying t-SNE, we obtain Figure 10(a), where the point dispersion demonstrates the diversity of environment themes. Figures 10(b) and (c) show the distribution of tool counts across environments, while Figure 10(d) shows the distribution of state categories. Figure 10(e) shows synthesized scenario counts per environment after removing unqualified scenario.

In Example D, we provide a program example of synthesized environments, and Table 11 shows the corresponding tool interfaces.

Table 9 present the statistics of each benchmark.

Table 10 shows the performance of Qwen3-4B training on different SFT datasets in non-thinking mode. We also compare the APIGen (Liu et al., 2024) and APIGen-MT (Prabhakar et al., 2025) datasets because they only provide non-thinking samples. APIGen contains 60K single-turn tool-use examples; APIGen-MT contains 5K multi-turn examples, but they cover only two environments, which are identical to those in Tau-Bench. For EnvScaler, we remove the teacher LLM’s reasoning process and keep only the action sequences. First, APIGen leads to degraded performance on all benchmarks, indicating that relying solely on single-turn supervision cannot improve an LLM’s multi-turn tool-use capabilities. Due to the environments being the same as in Tau-Bench, APIGen-MT provides a significant boost on Tau-Bench, but it fails to transfer to BFCL-MT or ACEBench-Agent. Moreover, since this dataset uses only the Conversation setting, the model tends to talk to the user before using tools, which conflicts with the style of BFCL-MT. This highlights the importance of training in both settings. Finally, EnvScaler (SFT) brings significant improvements on BFCL-MT and ACEBench-Agent but slightly degrades performance on Tau-Bench. We believe this is because Tau-Bench relies more on reasoning, while in non-thinking mode the LLM learns only actions and not the reasoning chain, making it difficult to produce correct actions during evaluation.

As stated in Section 3.4, we conducted 100 rounds of cyclic testing for each synthetic environment and retained only those with a test success rate above 0.85. From 266 synthetic environments, we selected 191, corresponding to a rejection rate of 28.2%. In Figure 11 (a), we present the distribution of scores for all environments in the dual-agent loop assessment. To systematically analyze the reasons for environment failures, we extract all test cases identified as failures from the logs in the dual-agent loop quality inspection. We use Claude-4.5-Sonnet to classify these errors. As shown in Figure 11 (b), the primary issue is type errors or attribute errors, followed by the consistency of state management.