Qiushi Sun   Zhoumianze Liu   Chang Ma   Zichen Ding   Fangzhi Xu  
Zhangyue Yin   Haiteng Zhao   Zhengyu Wu   Kanzhi Cheng   Zhaoyang Liu  
Qintong Li   Jianing Wang   Robert Tang   Tianbao Xie   Xiaochong Feng
Xiang Li   Ben Kao   Wenhai Wang   Biqing Qi   Lingpeng Kong   Zhiyong Wu

We introduce ScienceBoard, a realistic and multimodal environment designed to evaluate and advance computer-using agents for scientific discovery. By integrating domain-specific software and curating a benchmark of validated workflows, ScienceBoard enables rigorous assessment of agents’ abilities to operate in real scientific settings.

OS-Genesis is built around the following components, aiming for synthesizing high-quality trajectory data for GUI agents:

1. §Computer-using Agents for Scientific Discovery: ScienceBoard pioneers the application of computer-using agents to real-world scientific discovery, enabling digital automation for complex science tasks.
2. §Infra for Scientific Discovery: ScienceBoard provides a visually rich and dynamically configurable Ubuntu-based VM serves as the scientific playground, supporting both GUI and CLI interactions and enabling agents to perform end-to-end tasks like simulation and computation.
3. §Challenging Benchmark: We curate a benchmark of 169 human-annotated tasks across 6 scientific domains, each paired with automated evaluation functions for reliable task validation.
4. §Systematic Evaluation and Analysis: Comprehensive experiments are conducted with state-of-the-art LLMs, VLMs, and GUI action models. We further provide insights into current agent limitations and guiding future research.

Main
Pipeline Evaluation
Results In-depth
Analysis Code
Data

Click to jump to each section.

We will release all codes for infra, benchmark, evaluation pipelines and more details. We hope ScienceBoard can inspire and boost future research advancing computer-using agents in scientific workflows.

Computer-using Agents for Science

Training with high-quality GUI trajectories is essential for enhancing agentic capabilities. Ideal GUI agent trajectories include the following key components:

1. a high-level instruction that defines the overall goal the agent aims to accomplish
2. a series of low-level instructions that each describe specific steps required
3. actions (e.g., CLICK, TYPE)
4. states, which include visual representations like screenshots and textual representations such as a11ytree

ScienceBoard Environment

After reverse task synthesis generates high-level and low-level task instructions, these instructions are executed within the GUI environment to create complete trajectories.

To ensure the quality and utility of these trajectories, OS-Genesis employs a Trajectory Reward Model (TRM). Built upon GPT-4o, TRM evaluates each trajectory based on completion (task fulfillment) and coherence (logical sequence of actions), assigning a graded reward score from 1 to 5. Unlike traditional binary filtering methods, TRM allows even incomplete but valuable trajectories to contribute to training.

ScienceBoard Benchmark

After reverse task synthesis generates high-level and low-level task instructions, these instructions are executed within the GUI environment to create complete trajectories.

pie chart and table here

To ensure the quality and utility of these trajectories, OS-Genesis employs a Trajectory Reward Model (TRM). Built upon GPT-4o, TRM evaluates each trajectory based on completion (task fulfillment) and coherence (logical sequence of actions), assigning a graded reward score from 1 to 5. Unlike traditional binary filtering methods, TRM allows even incomplete but valuable trajectories to contribute to training.

---

Evaluations

Main Settings

We first evaluate OS-Genesis on mobiles tasks, covering AndroidWorld (in-domain setting) and AndroidControl (OOD setting). The results are shown in Table 1

chart here

AndroidWorld: OS-Genesis demonstrates exceptional performance on the AndroidWorld benchmark, significantly narrowing the gap between open-source agents and the state-of-the-art GPT-4o-based M3A agent. Training with OS-Genesis-synthesized data achieves nearly double the success rates compared to task-driven methods, with a success rate improvement from 9.82% to 17.41% for Qwen2-VL-7B and substantial gains for other backbones like InternVL2-8B.

AndroidControl: On AndroidControl, OS-Genesis showcases strong OOD capability, outperforming baselines in both high/low-level tasks despite encountering only 20 of 833 apps during synthesis. It achieves superior action and planning, validating our exploration-first approach for generating diverse, high-quality tasks and adapting effectively to unseen environments.

Disentangled Planning and Action

Then, we evaluate OS-Genesis on web task, using challenging online benchmark: WebArena as the testbed. The results are shown in Table 2

WebArena: On WebArena, OS-Genesis delivers notable performance improvements across diverse 5 navigation scenarios, outperforming task-driven baselines and achieving significant gains with InternVL2-8B and Qwen2-VL-7B backbones. By leveraging reverse task synthesis, OS-Genesis effectively explores the rich interactive elements of web environments, producing more meaningful and diverse trajectories.

Chart here, GPT4o + x

Analysis

How Scaling Trajectory Data Improves Agentic Ability?

We investigate the impact of data scale on building GUI agents. To explore this, we partition the data synthesized by OS-Genesis into subsets, ranging from small-scale trajectories to those exceeding the size used in main experiments. Using AndroidWorld as our testbed, we focus on two primary questions: (1) How does performance improve as the data scale increases? (2) Does performance saturate at higher data scales?

As shown above, task performance generally improves as the number of trajectories increases, while saturation emerges at larger data scales.

How Far are we from Human Data?

We also investigate the gap between OS-Genesis-synthesized data and human-annotated data. (1) trajectories from OS-Genesis v.s. human-annotated trajectories. We select 1K crowdsourced trajectories from AndroidControl training set for comparison. As shown below, we significantly narrow the performance gap between synthetic trajectories and human-annotated trajectories. This is notably evident in high-level tasks, demonstrating that agents trained on OS-Genesis trajectories can plan and solve problems more closely aligned with human manners. In terms of average success rate, viewing human-annotated data as the gold standard, the performance retention rate of OS-Genesis data surpasses 80%.

(2) high-level instructions synthesized through OS-Genesis v.s. human-written instructions. For comparison, we match 500 human-written tasks from the AndroidControl training set and use GPT-4o for exploration. As observed, even when high-level instructions are written by human, their performance falls short compared to OS-Genesis's instructions. This can be attributed to two main factors: (a) Pre-defined tasks sometimes fail to align with the dynamic environment, and (b) Models may introduce errors when interpreting the intentions of human annotators. In contrast, OS-Genesis generates data in a progressive way, grounded in low-level interactions, which makes it inherently more suitable for unsupervised exploration and adaptation.

Screenshot A11y Tree Screenshot + A11y Tree Set of Marks

Conclusion

OS-Genesis is a data synthesis pipeline designed to revolutionize the construction of GUI agent trajectories. Reverse task synthesis enables the generation of diverse and coherent tasks by retroactively deriving instructions from observed interactions, while TRM ensures the quality of trajectories through graded evaluations. Together, OS-Genesis address critical challenges in GUI agent trajectories construction, paving the way for high-quality agentic data generation. We hope that it can provide a promising direction for generating high-quality trajectory data for GUI agents, bringing the community one step closer to achieving digital automation.

Acknowledgement

We would like to thank OSWorld authors for helping us tackle various issues in building infra and task evaluation, as well as the Cambrian authors for providing this webpage template.