UI-Venus-1.5 Technical Report

The Mid-Training phase is designed to bridge the semantic gap between general visual perception and the fine-grained structural understanding required by GUI Agents. Specifically, general-purpose vision-language models often lack the granularity needed to capture the structural nuances of user interfaces. In the subsequent reinforcement learning phase, this deficiency severely hinders effective exploration, leading to sparse reward signals and preventing policy improvement from bootstrapping in complex interaction scenarios. This limitation is mainly due to the scarcity of GUI-specific structural modeling in standard pre-training corpora. Consequently, rather than relying solely on capability elicitation, we shift our objective toward foundational representation building. This phase (Mid-Training) enables the model to encode diverse GUI layouts and interaction logic, providing a robust initialization for subsequent policy optimization, which can also be experimentally verified in Section 3.3.

To support the Mid-Training phase, we constructed a unified corpus by aggregating over 30 diverse sources, including Mind2Web Deng et al. (2023), ShowUI Lin et al. (2024), AITW Rawles et al. (2023) and so on. The hierarchical distribution of this corpus is detailed in Figure 4(a). This 10B-token dataset is strategically stratified to ensure functional diversity: semantic perception (20.8%) and GUI-VQA (22.1%) provide the representational foundation, while grounding (24.8%) and hybrid navigation-reasoning tasks ensure execution robustness.

Based on this unified corpus, the Mid-Training data supports four complementary supervision objectives covering perception, reasoning, and action alignment. Specifically, the dataset provides supervisions for:

With the large scale data we collected for Mid-Training, the next step is to clean and refine the low-quality navigation traces since some open-source datasets often contain noise that can limit performance gains. To address this, we propose a teacher-based quality refinement module to rank, select, and rewrite the Mid-Training data. Specifically, we first utilize Qwen3-VL-235B-A22B Bai et al. (2025a) to evaluate and rank the input data with numerical quality assessments as illustrated in Figure 4(b) following the LLM-as-a-judge fashion. We chose this model because of its superior generalization and reasoning capabilities among open-source models. Mid-Training samples are scored from 0 to 10 based on action-visual alignment and task reachability according to our carefully designed prompts. Among the results, high-quality traces(score $\geq 7$) are retained in gold pool since the goal is accomplished; mid-quality traces(4-6) are routed to a rewriting model to refine the instruction according to the last state; and low-quality samples(0-3) are totally reconstructed or just discarded.

By performing recursive refinements, the proportion of high-fidelity samples in our dataset increased from 69.7% to 89.7%. Finally, we conduct subsequent manual verification of sampled trajectories from the gold pool and ensure that the training signal remains both dense and accurate.

To further improve robustness in real-world environments, we augment the Mid-Training corpus with interaction trajectories collected from real-device execution. Compared with static open-source datasets, real-device interaction data better captures execution failures, GUI dynamics, and environment-dependent behaviors that GUI agents must handle in practice. We therefore build a data generation loop on top of our DaaS system (Section 2.3.2). As illustrated in Figure 5, an open-source MLLM first generates candidate task prompts from seed instructions. After semantic verification based on embedding similarity, valid and non-duplicate prompts are stored in a task bank and executed by GUI agents on cloud-hosted devices. The system then performs GUI trajectory scraping followed by multi-annotator verification to collect high-quality interaction trajectories. A key feature of this pipeline is its iterative generation loop. Verified trajectories are fed back to the MLLM as in-context examples for subsequent task generation, enabling progressively more executable and realistic task prompts. As a result, the success rate of the trajectory generation pipeline improves from 17.9% to over 70% after several iterations. In total, this real-device generation loop produces more than 30,000 verified interaction trajectories.

Reward: Following our previous work, the reward function is composed of two components formally: We first check whether the predicted answer string conforms to a predefined syntax as the format reward $R_{\text{format}}$. Next, given a screenshot and the instruction, the model must predict a center point that localizes the element. We use the commonly used point-in-box reward noted as $R_{\text{point-in-box}}$ to train the model.

Combining all components, the final action-wise reward is computed as:

where $w_{1}$ and $w_{2}$ control the relative importance of format correctness and location precision.

Refusal Samples: A major departure from UI-Venus-1.0 is the modification of our grounding prompts to incorporate what we define as refusal capability. Specifically, when faced with an instruction that refers to an element or icon, not present in the image, the model is trained to return a fixed output of $[-1,-1]$, refer to our prompt as shown in A.2. Compared to models that strictly output coordinates regardless of the instruction’s validity, this refusal-aware approach is significantly more intelligent and effectively mitigates hallucinations during user interactions.

Although the introduction of refusal prompts may lead to a marginal performance trade-off on benchmarks lacking refusal examples (such as ScreenSpot-Pro Li et al. (2025a)), UI-Venus-1.5 nevertheless maintains its state-of-the-art (SOTA) standing. Furthermore, on benchmarks that explicitly include refusal tasks—such as VenusBench-GD Zhou et al. (2025a), and OSWorld-G-Refine Xie et al. (2025b), our model achieves new SOTA results, particularly demonstrating superior accuracy on refusal-specific samples.

In this section, we detail the Offline-RL formulation and empirical observations across Mobile and Web navigation tasks. In reinforcement learning, a well-designed and robust reward system is essential for stable policy optimization. Despite the differences between Mobile and Web platforms, GUI agents typically share a substantially overlapping action space allowing for a unified reward design across both domains.

Reward: To ensure the agent model generates structured, executable outputs, we build upon our prior UI-Venus-1.0 Gu et al. (2025) framework and adopt a decoupled reward system comprising two primary components: (i) a format reward and (ii) an action reward.

Compared to the UI-Venus-1.0 baseline, we enhance navigation performance in web environments by introducing several specialized GUI actions, such as Hover and Hotkey. Furthermore, we implement domain-specific constraints for the Scroll action to align with the different operational logics of each platform: in mobile tasks, the model must predict precise start and end coordinates, whereas in web tasks, it is only required to specify the scrolling direction.

Overall, the total reward $R$ is formulated as:

where $w_{1}$ and $w_{2}$ control the trade-off between the format reward and the action reward.

Discrepancy Between Step and Trace Success Rates: We use the above reward system for stable Offline-RL. However, we observe a notable trend during Offline-RL training. As shown in Figure 6, while step-level success rates increase steadily, trace-level success rates eventually peak and then decline. We attribute this behavior to an inherent limitation of Offline-RL: step-level rewards only optimize individual actions and fail to guide the successful composition of a full multi-step trace. To improve the deployable performance of the model, we therefore append an online reinforcement learning stage after the Offline-RL, explicitly optimizing for trace-level rewards and substantially enhancing the model’s end-to-end task completion.

While SFT and Offline-RL provide a solid initialization for GUI agents, their effectiveness is inherently constrained by existing static datasets and predefined interaction distributions. In real-world GUI environments, agents are required to navigate dynamic GUI states, stochastic system behaviors, and extended decision-making horizons, where real-time feedback during execution is critical to performance. Pure offline training is insufficient to address these challenges. Online Reinforcement Learning(Online-RL) responds to these limitations by enabling the agent to learn and adapt in real time through direct interaction with the environment. This allows for rich and immediate feedback and better handling of uncertainties in dynamic settings especially in long interaction sequences. Consequently, policies trained solely offline often exhibit brittleness when deployed in novel GUI layouts or unexpected intermediate states. These limitations are further corroborated by the inconsistencies in step-trace accuracy observed in Section 2.2.

To address these limitations, we introduce online reinforcement learning as a complementary training paradigm. By enabling direct environmental interaction, Online-RL enables the agent to collect trajectories that reflect the actual deployment distribution and to iteratively refine and update its policy based on observed feedback. This leads to the design of a dedicated online learning framework, which comprises a robust execution infrastructure called DaaS (Section 2.3.2), comprehensive task generation and reward formulations (Section 2.3.3), and a stable RL training algorithm (Section 2.3.4).

Training and deploying a GUI Agent capable of generalizing across heterogeneous devices imposes stringent demands on the underlying execution environment in terms of uniformity, extensibility, and performance. Unlike simulators designed for a single device category, such an environment must accommodate a wide variety of device types, offer a unified abstraction over diverse interaction protocols, and—under strict network-isolation policies, securely and efficiently expose large-scale device resources to upstream training and deployment frameworks. Therefore, we build a unified Device-as-a-Service (DaaS) layer (Figure 7) to meet these requirements. It consists of two core components: the Group Control Gateway (GCGW) and the Unified Client SDK, whose design and implementation are detailed next.

Group Control Gateway (GCGW). The GCGW serves as a high-performance centralized reverse proxy and the orchestration core of the DaaS layer. It abstracts heterogeneous control protocols across diverse platforms—including Android (ADB), Chrome (CDP), and Linux containers (SSH)—enabling extensible support through a protocol-centric abstraction.

To ensure system stability and performance under high-throughput workloads, the GCGW integrates several key architectural optimizations. First, to handle stateful protocols like ADB and CDP which rely on long-lived connections, the GCGW employs an internal secondary hash routing algorithm. This mechanism ensures that all requests for a specific device are consistently routed to the same gateway node, effectively preventing the “$M\times N$ connection explosion” problem($M$ gateway nodes and $N$ devices) and significantly reducing the connection overhead per node. To support this routing strategy without sacrificing performance, the gateway utilizes streaming transmission and zero-copy I/O for internal forwarding between nodes, ensuring near-zero additional latency. Furthermore, the entire GCGW architecture is built on a high-concurrency coroutine model. This design is specifically tailored for the “high-concurrency, low-frequency” access patterns of device operations, allowing the system to maintain hundreds of thousands of concurrent connections with minimal memory overhead.

Unified Client SDK. To further shield downstream users from the complexities of infrastructure management, a multi-language Unified Client SDK was developed that acts as a high-level API layer atop the GCGW. This SDK encapsulates several critical functions into a streamlined workflow. Specifically, it automates device lifecycle management, including device preemption, heartbeat maintenance, and resource release. Furthermore, it provides a unified semantic interaction interface that standardizes communication across various internal protocols. By abstracting these low-level operations, the SDK significantly lowers integration barriers, allowing downstream teams to concentrate on building and operating the end-to-end training pipeline for large-scale online reinforcement learning of GUI-specialized models, as well as the production deployment of those GUI-focused models—rather than dealing with protocol-specific technicalities.

By implementing these architectural optimizations, the DaaS layer achieves the following performance benchmarks:

Ahead of the training principles and pipeline of online-RL, we introduce task formulation and reward design which are fundamental to the success of online-RL. The quality, diversity, and calibrated difficulty of the input tasks define the potential ceiling of policy optimization, while the specialized reward function serves as the primary driver for training efficiency and stability.

Task Generation and Stratified Sampling: The performance ceiling of online-RL is fundamentally governed by the diversity and quality of its task pool $\mathcal{T}$. To this end, we employ a hybrid generation strategy combining static heuristics with dynamic evolution:

Reward: To guide the agent effectively in complex GUI environments, we design a composite reward function $R$. For a given execution trajectory $\tau=(a_{0},a_{1},\dots,a_{T})$ with $T$ steps, the total reward consists of a task completion reward $R_{comp}$, an action constraint penalty $R_{p}$, and a trace length decay coefficient $\eta\in(0,1]$:

where $T_{\text{min}}$ is the minimum number of steps to success among a group of trajectories collected during the online RL process. Specifically, to encourage the agent to learn the shortest operational path, we introduce a decay coefficient $\eta$. A larger step count $T$ results in a lower final reward, thereby suppressing redundant or circular actions during policy gradient optimization. Another important factor of GUI Agent is to generate the correct actions in the predefined action pool, i.e., the agent’s output must adhere to specific syntactic specifications. If the agent’s generated response cannot be recognized by the parser as a valid action, a negative penalty $\lambda$ is assigned at the current step. The penalty term is defined as:

By incorporating $R_{p}$, we significantly reduce the number of invalid attempts during the online exploration phase, thereby improving sample efficiency.

After that, we implement a dual-track verification mechanism to determine task success ($\mathbbm{1}_{success}$):

We employ the Group Relative Policy Optimization (GRPO) algorithm. Unlike the conventional Actor-Critic framework, GRPO estimates relative advantages directly from a group of sampled trajectories, thereby bypassing the need for a separate value function network. This approach substantially reduces computational complexity and effectively addresses the convergence challenges posed by sparse reward signals in GUI-based tasks.

In each training step, for a task $q$ sampled from the task pool, the agent generates a group of $G$ complete interaction trajectories $\{\tau_{i}\}_{i=1}^{G}$ using the current policy $\pi_{\theta_{old}}$. The GRPO loss function $L_{GRPO}(\theta)$ is defined as follows:

where $r_{i,t}(\theta)=\frac{\pi_{\theta}(a_{i,t}|s_{i,t},q)}{\pi_{\theta_{old}}(a_{i,t}|s_{i,t},q)}$ denotes the importance sampling ratio between the new and old policies at the action (or token) level.

Trajectory-level Advantage Calculation and Assignment: Given the long execution horizons of GUI tasks and the difficulty in identifying critical actions, we forgo step-wise reward signals in favor of evaluating the overall quality of each complete trajectory $\tau_{i}$. The trajectory-level advantage $\hat{A}_{i}$ is derived by normalizing relative scores within the sampled group:

where $R(\tau_{i},q)$ is the composite reward defined in Section 2.3.3 (incorporating task success rewards and invalid action penalties). The calculated $\hat{A}_{i}$ is uniformly assigned to all action steps within the trajectory. By relying on competition within the sampled group, this approach filters out environmental stochasticity and supplies a stable credit assignment signal, thereby supporting stable policy updates in long-horizon decision-making.

Training Stability and Exploration Enhancement: Rewards in GUI navigation are both sparse and costly to verify, which can lead to policy collapse during extended online training. To mitigate these issues, we implement two complementary regularization mechanisms:

The final total optimization objective for the Online-RL stage is:

UI-Venus-1.5 incorporates the Qwen3-VL Bai et al. (2025a) architecture as its core backbone, leveraging its advanced multimodal processing capabilities to interpret complex visual interfaces. Note that we map all spatial targets into a normalized $[0,1000]$ coordinate space following Qwen3-VL and unifies diverse events into a shared set of action space as shown in Table 8.

Note that methods marked with * indicate our own reproductions with official prompts, some of which achieve superior performance compared to their original reported results. Moreover, all methods evaluated in this section follow the end-to-end design without any zoom-in or agent-framework strategies except specially mentioned.

Grounding: We evaluate the model’s grounding capabilities with seven complementary benchmarks: VenusBench-GD Zhou et al. (2025a) for its assessment of high-level reasoning and refusal capabilities, ScreenSpot-Pro Wu et al. (2024) focuses on high-resolution, fine-grained professional layouts, UI-Vision Nayak et al. (2025) assesses reasoning abilities (e.g., spatial and functional) in diverse applications, MMBench-GUI L2 Wang et al. (2025c) evaluates hierarchical-instruction following and compositional reasoning, OSWorldG and OSWorld-G Refine Xie et al. (2025b) jointly assess comprehensive skills such as layout understanding, widget matching, and fine-grained manipulation, and ScreenSpot-V2 Wu et al. (2024) serves to broaden coverage across different operating systems.

Navigation: We first evaluated UI-Venus-1.5 on two widely-adopted online mobile benchmarks: AndroidWorld Rawles et al. (2025) and Androidlab Xu et al. (2025), based on a live Android emulator with various applications and tasks. In addition, we also performed experiments on VenusBench-Mobile, a more challenging benchmark whose tasks are more related to real-world human applications. To further validate the cross-domain versatility of UI-Venus-1.5, we extended our evaluation to real-world web environments. We evaluate our model on a representative subset of WebVoyager He et al. (2024), as the full benchmark is time-consuming to evaluate and some time-sensitive tasks are no longer valid.

In the experiments, we compare UI-Venus-1.5 models against various state-of-the-art baselines across two different model categories: (1) General VLMs: Seed1.8 Seed (2025a) and widely used Qwen3-VL Bai et al. (2025a). (2) GUI-specific Models: OpenCUA Wang et al. (2025b), UI-TARS-1.5 Seed (2025b), GTA1 Yang et al. (2025), Gui-Owl Ye et al. (2025), Holo2 H-Company (2025), Step-GUI Yan et al. (2025) and MAI-UI Zhou et al. (2025b).

VenusBench-GD. VenusBench-GD is a comprehensive grounding benchmark spanning web, desktop, and mobile UIs, covering both basic localization and advanced, reasoning-heavy cases, and further includes refusal grounding to test whether agents can correctly reject infeasible instructions. As shown in Table 1, UI-Venus-1.5 achieves strong and scalable performance, with our 30B-A3B model reaching 75.0% and outperforming competitive GUI-specialized baselines.

ScreenSpot-Pro. ScreenSpot-Pro focuses on high-resolution professional software interfaces (e.g., CAD, development tools, creative suites, and office applications), where dense layouts and small icons make fine-grained grounding particularly challenging. In Table 1, UI-Venus-1.5-30B-A3B achieves the best overall accuracy at 69.6%, exceeding the strongest reported baseline MAI-UI-32B (67.9%) and showing clear gains over smaller UI-Venus-1.5 variants (57.7% for 2B, 68.4% for 8B).

ScreenSpot-V2. ScreenSpot-V2 is a broad cross-platform grounding benchmark covering mobile, web, and desktop UIs with both text and icon/widget targets, reflecting everyday GUI interaction scenarios. As shown in Table 1, UI-Venus-1.5 achieves strong performance, with 30B-A3B reaching 96.2% (second-best, 0.3% behind MAI-UI-32B) and 8B reaching 95.9%, demonstrating robust generalization even in a near-saturated benchmark.

MMBench-GUI L2. MMBench-GUI L2 evaluates instruction-following grounding with both Basic (low-level attributes) and Advanced (goal-oriented, compositional) instructions, testing a model’s ability to align implicit user intent with the correct UI element. In Table 1, UI-Venus-1.5 remains competitive, reaching 88.6% with 30B-A3B.

OSWorld-G. OSWorld-G(and its refined split OSWorld-G-R) contains fine-grained desktop grounding tasks that require diverse skills such as text matching, widget recognition, layout understanding, and precise manipulation. Table 1 shows UI-Venus-1.5-30B-A3B achieves state-of-the-art performance on both settings, reaching 76.4% on OSWorld-G-R and 70.6% on OSWorld-G, surpassing strong baselines such as MAI-UI-32B (73.9% / 67.6%) and GTA1-32B (72.2% / 65.2%).

UI-Vision. UI-Vision is a license-permissive benchmark for desktop GUI grounding that emphasizes real-world applications and fine-grained reasoning (e.g., spatial and functional understanding). As shown in Table 1, UI-Venus-1.5-30B-A3B achieves the strongest result at 54.7%, substantially outperforming prior GUI-specific baselines (e.g., MAI-UI-32B at 47.1%).

In summary, our evaluation reveals several key insights:

In addition to evaluating GUI grounding capabilities, we further assess the UI-Venus-1.5 series on four navigation benchmarks spanning both mobile and web environments, including Android World Rawles et al. (2025), Android Lab Xu et al. (2025), VenusBench-Mobile and WebVoyager He et al. (2024). These benchmarks are challenging, fully dynamic online suites that require GUI agents to perform multi-turn adaptive perception, reasoning, and action in evolving environments, thus providing a more reliable assessment of the model’s navigation capabilities.

Android World. AndroidWorld is a comprehensive online evaluation environment for GUI agents. It includes 116 programmatic tasks across 20 real-world Android applications and is widely used as a benchmark for assessing GUI agent performance. As shown in Table 3, the Venus-1.5 model family achieves state-of-the-art (SOTA) performance among models of comparable scale. Specifically, our 2B / 8B / 30B-A3B variants reach accuracies of 55.6% / 73.7% / 77.6%, outperforming existing domain-specific and general-purpose baselines. Relative to the strongest existing contender, MAI-UI-32B, UI-Venus-1.5-30B-A3B secures an absolute margin of 4.3%.

Android Lab. Android Lab is another dynamic, online evaluation benchmark comprising 138 tasks across 9 Android applications. Our model uses only raw screen screenshots as input, yet it outperforms both a range of GUI-specialized models and general-purpose models, some of which take both screenshots and XML information as inputs. As shown in Table 3, our UI-Venus-1.5 series models (2B, 8B, and 30A3B) achieve 36.2%, 55.1%, and 52.9% on AndroidLab, respectively. It is worth noting that, due to bugs in the official AndroidLab evaluation code, we additionally report human-verified results (marked with a $\dagger$) in the Table 3. The corrected results show that our UI-Venus-1.5-30A3B model does not exhibit any performance degradation on AndroidLab compared with the UI-Venus-1.5-8B model (68.1% vs. 68.1%). Compared with our UI-Venus-1.0-72B model, the best model in the 1.5 series yields up to 5.8% improvement. Notably, even UI-Venus-1.5-8B significantly outperforms other state-of-the-art models.

VenusBench-Mobile. VenusBench-Mobile is a challenging benchmark designed to evaluate the end-to-end performance of GUI agents in complex mobile environments. As illustrated in Table 5, the UI-Venus-1.5 model family achieves state-of-the-art (SOTA) performance across all scales. Specifically, our 2B, 8B, and 30B-A3B variants reach success rates of 8.7%, 16.1%, and 21.5%, respectively, consistently outperforming both general-purpose VLMs and specialized GUI models. Notably, our 8B model (16.1%) already surpasses the much larger UI-Venus-1.0-72B (15.4%), while our 30B-A3B variant sets a new record with a substantial 6.1% absolute margin over the previous best-performing model.

WebVoyager. WebVoyager is a comprehensive end-to-end benchmark for evaluating web agents on 15 real-world websites including e-commerce, travel, and social platforms. It employs an automatic evaluation protocol leveraging MLLM to assess task completion rates, measuring agents’ abilities to autonomously navigate and interact with dynamic web environments through visual screenshots and textual elements. As shown in Table 5, the UI-Venus-1.5 model family achieves comparable performance among models of comparable scale. Specifically, our 2B/8B/30B-A3B variants reach success rates of 56.4%/70.8%/76.0%, outperforming WebVoyager He et al. (2024) and general VLMs (e.g., GPT-4o, Qwen3-VL).

In summary, our evaluation reveals several key insights: