Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
multi-modal agent
We thank the reviewer for the valuable feedback. We are encouraged that the reviewer finds our benchmark to complement existing works, and provides a useful parallelization infrastructure. We address each question/concern below:

**Additional benchmark improvements**
- We agree that benchmark additions like multilingual tasks, improved metrics (e.g. cost, steps, time to completion) can improve agent evaluation. We note that our current tasks and metrics provide ample coverage across the most utilized Windows functions in terms of representative user workflows. Beyond our current analyses of human evaluation (Appendix A.5) like number of steps, difficulty ratings, and agent/task job runtimes (Appendix A.7, Table 11), we are happy to include additional details. 
- We agree; the points raised by the reviewer are improvements that we plan to make over time on the open-sourced codebase nonetheless; this includes improving benchmark tasks in quantity and quality (with a focus on multi-application tasks), adding new applications/domains, new modalities to the benchmark (e.g., audio-capable agents), and adding Chinese, Japanese, and Spanish versions of tasks to our benchmark. 
  - Given that our goal is to first make the benchmark scalable/parallelizable to enable faster evaluation/reiteration, our planned improvements can be easily added atop our framework, along with other features, as we co-develop our benchmark with the research community on Github—especially as model/agentic capabilities evolve.
 
**Comparison with other agents**
- Our main goals with Windows Agent Arena are to provide a new benchmark for Windows OS and offer an accessible, preliminary agent (Navi) which researchers can adapt for their own needs. Navi is based on set-of-mark prompting and VLM reasoning, which is a common agent architecture. 
- There is a rich body of literature with multiple agent types, but implementing all these variants was out of scope given our main focus is on the construction of a scalable, parallelizable benchmark for the Windows OS environment. However, we still make a best effort to ablatively study our agent's capabilities as we compare 32 different Navi variants (Table 4) using multiple screen parsing models and VLMs (from small to large models) to help users understand the pros/cons of each configuration. 
  - Our intention for Navi is to illustrate the use of our benchmark and serve as an accessible but relatively capable starting point/template for the community. Together with our benchmark, our hope is that our contribution will enable faster reiteration in terms of research and development of AI agents.

**[Q1] What are our contributions / How is this different from OSWorld?**
- Please see **[N2]** in "General Response to Common Questions from Reviewers" at the top. 

**[Q2] Failure analysis**
- Section 4.2 details common failure causes for Navi: i) visual-language misalignment (selecting the wrong element ID despite a reasonable textual explanation), and ii) incorrect screen parsing (e.g. grouping multiple elements under the same ID). 
- Figure 6 provides visual examples of these failures. Appendix B (Figure 16) provides another example. We are happy to include more examples in the appendix in the revised manuscript. We hope these examples help researchers when developing their own agents.

**[Q3] Comparison with OSWorld’s agent**
- OSWorld's agent uses as input the accessibility (a11y) tree along with the raw screenshot. Navi has a similar implementation, but instead uses the Windows UIA tree and other models (e.g., Omniparser) to create SoMs on the screenshot to aid element selection. 
- Moreover, several of our agent tests/choices already share substantial overlap with OSWorld’s including GPT-4V (which delivered the best on-average performance in OSWorld’s experiments and received the most analysis). 
- We also include new models released since OSWorld (e.g., OpenAI’s o1 for planning/reasoning or Omniparser for screen parsing) as well as other types of models not covered by OSWorld or other benchmarks (e.g., small language models like phi3). 
- Overall, we study and analyze 32 variations of Navi (using different screen parsing backbones and VLMs for reasoning/planning).

**[Q4] Human performance**
- Appendix A.5 provides details on human evaluation/performance. 
- Participants operated without search tools or the internet; failures were mostly due to not knowing the lengthy steps needed to change specific settings in media players and office apps. In several cases, participants were not as familiar with the program/application (e.g., more familiar with another media player). In others, the participants’ typical workflows were more casual which did not require advanced use cases in their day-to-day tasks compared to what the task required. Our tasks intentionally reflect a range of difficulty levels. 

Please let us know if there are any other questions and we would be happy to address them!
We thank the reviewer for the valuable feedback. We are encouraged that the reviewer finds that our benchmark is complementary to existing works in the field, and adds a more efficient evaluation. Below, we address each concern in turn:

**Navi’s novelty and Navi’s role in our paper**
- To clarify, we do not claim in this work that Navi is an entirely new type of agent. Yes, as noted by the reviewer, our goal/aim for Navi (and its variants) was to have it serve as a reasonable, capable, but preliminary starting point and baseline for users, researchers, and the open-source community at large to develop their own agents using our new benchmark. 
- Together with our novel contribution in making this benchmark scalable and parallelizable, we aim to provide an accessible way for the community to train/test agents and reiterate efficiently. 
- Navi follows similar approaches to other web / OS agents which use screen parsing + VLM reasoning. In Table 4 and in the results section of our paper, we provide a thorough analysis of 32 different combinations of Navi agents with distinct backbone models, which can help guide researchers in this area to develop and customize their own solutions. 

- We have also open-sourced these backbones to allow researchers to create their own agents which might contain better solutions and move the benchmark forward. In addition, we benchmark Navi on a more established, “traditional” agent benchmark (Mind2web) to verify its performance. 
- Currently, we do not plan on testing Navi on other benchmarks for this paper since our main goal for this paper is not to make Navi competitive. Rather, our paper’s main contribution and novelty are developing a Windows OS benchmark that is scalable through its parallelization infrastructure (for fast evaluation) and realistic for training/testing agents in a Windows environment. 
- To demonstrate the promise of agent development and help showcase the benchmark, Navi serves as both a beginning baseline and openly available template for users to play with and develop in this respect.

We hope our responses have addressed your questions. Please let us know if there are any other questions or suggestions and we would be happy to address them.
We thank the reviewer for the valuable feedback. We are encouraged that the reviewer finds that our benchmark fills a gap in current agent evaluations. Below we attempt to address each of your concerns within the space limit.

**Accessibility and proprietary OS**
- We are aware of this limitation which is why we have made significant effort so that anyone can access/use Windows OS in our benchmark; despite not being an open-source OS, our code repository enables users on to access a free evaluation copy which can be used for benchmark deployment with easy continued renewal after. 
- To the best of our knowledge, our benchmark is one of the few that provides users open/free access to the Windows OS and computing environment for research/development. Please see **[N1]** in "General Response to Common Questions from Reviewers" Official Comment at the top.

**Resource-intensiveness**
- See **(2)** of **[N2]** under "General Response to Common Questions from Reviewers" Official Comment. 
- In short, our benchmark does not need any high-performance computing setups and large storage capacities. 
- After each run, the internal state of the system/VM is reset to ensure that progress/completion on one task does not interfere with others. This preserves the integrity of the OS environment so tasks can be continuously run and parallelized without interruption or issue. This also removes all created files etc. resulting from agent action, further mitigating the need for a large storage capacity. 

**Configuration/maintenance**
- Our benchmark and the underlying infrastructure make it easy to install additional software either on the VM directly or via the Docker image. This way, users can design new tasks that require new programs and add them to the benchmark as necessary. 
- Together with the ease in which a user can design their own tasks through a task configuration file/JSON (e.g., see README and Appendix A.6), our benchmark is easily configurable, flexible, and extensible to new tasks, programs, etc., with minimal risk of breaking the benchmark's functionality or integrity. 
  - In defining their own custom tasks, users can leverage our existing library of configuration or evaluation functions or write their own by modeling theirs after ours. We are happy to include even more detailed instructions on task creation, hosting, and configuration beyond what we already have. 
- Unfortunately, many agentic benchmarks with computer/OS environments (e.g., OSWorld, AndroidWorld, etc.) share similar weaknesses when it comes to maintenance: as OS systems undergo refreshes, updates, etc., it can be difficult to ensure continued compatibility with the benchmark. 
  - To mitigate this, we have frozen automatic update and instead conduct manual review/updates (e.g., monthly) to re-adjust and mitigate things breaking. Since we are adding new features, modalities, and tasks, we will be continuously maintaining the benchmark in terms of both our needs and those of the wider community (e.g., Github).

**Ease of benchmark infrastructure**
- A major contribution of our benchmark is precisely making it easy for researchers to scale up agent experiments. As such, we have made significant effort to automatically build/install all necessary programs, scripts, images, etc. to simplify evals. We also include resources to help add/configure new tasks. 
- Cost-wise, a full parallel run costs in the range of 10 USD in total with CPU machines in Azure. Model calls cost about $100-200 depending on tokens utilized and model choice. We will include these details in our revised manuscript alongside data we already report (Table 10 in Appendix A.7).

**Windows-specificity / Why only Windows?**
- Please see **[N1]** in the comments we made for all reviewers.

**What are our contributions / How is this different from OSWorld?**
- Please see **[N2]**.

**Tasks**
- Please see **[N1]**.

**Navi’s novelty**
- To clarify, we do not claim that Navi is an entirely new type of agent. 
- We aim for Navi to serve as a reasonable starting point and baseline for researchers developing their own agents. 
  - As pointed out, it follows similar approaches as other web / OS agents which use screen parsing + VLM reasoning. 
- What we provide in Table 4 and in the results section is a thorough analysis of 32 different combinations of Navi agents with distinct backbone models to help guide researchers in this area. 
  - Out of those models we also open-source a few of these backbones to allow researchers to create their own agents which might contain better solutions and move the benchmark forward. We also benchmark Navi on a more well-established “traditional” agent benchmark (Mind2web) to verify its performance.

**Reward structure**
- We believe that the completion-based reward function is a reasonable assumption for the benchmark. However, we do plan to add number of steps, time, and cost as auxiliary metrics to the benchmark in our open-sourced codebase.
We thank the reviewer for the valuable feedback and are encouraged that the reviewer finds our benchmark to be well-designed and diverse. We address each concern below in turn:

**Why only Windows?**
- Please refer to note **[N1]** in the "General Response to Common Questions from Reviewers" Official Comment at the top.

**What are our contributions / How is this different from OSWorld?**
- Please refer to note **[N2]** in the "General Response to Common Questions from Reviewers".

**Extension to other OS**
- Generally speaking, there are two aspects/issues to consider if a researcher wants to extend our benchmark to another OS:
  - Task configurations/evaluation scripts, and benchmark parallelization infrastructure. While OSWorld already, and primarily, covers tasks with Linux OS in mind, it is not yet equipped with fast and scalable parallel evaluation in the cloud. Our approach/environment not only provides benefits when it comes to scalability and faster/parallelizable evaluation but also a more secure environment as well. 
  - Furthermore, OSWorld, to the best of our knowledge, does not natively support a full version of Windows OS to the extent that we do. A developer would be able to borrow infrastructure from Windows Agent Arena, but that project is outside of the scope of our paper. Porting the benchmark to MacOS would require both very heavy engineering work for porting tasks and infrastructure due to the structural differences between OS architectures.

**Uniqueness of tasks and capability assessment**
- We refer to **[N1]** for more detail. In short, because Windows OS has distinct UI, system-specific functions and command line, system-specific apps, idiosyncrasies, etc. than Linux OS, we expect agentic behavior, performance, and perception to vary considerably---as a result, the same task can have vastly different completion trajectories depending on OS. 
  - As just one example, UIA trees that are used as input (in addition to screenshots and SoMs) to help multi-modal agents understand/access UI elements on the screen exist on Windows; however, different structures exist on Linux, like AP-STI or ATK among others, which naturally will impact agent performance in how well it traverses these structures to perceive different UI/visual elements, etc. These will ultimately impact how the agent plans, reasons, and acts across different OS. We will clarify these aspects in the revised manuscript.
- Only 10% of OSWorld's tasks are for Windows (43 out of 412) whereas the entirety of our tasks (150+) focus on Windows. Moreover, we include a considerable portion of newly designed tasks in addition to tasks we converted or transformed to suit Windows function and usage. We will add more discussion re: these differences in our manuscript.
- Rather than focusing on Navi's evaluation across different benchmarks, our main goal is to provide a new benchmark for Windows OS, and offer an accessible, well-performing but preliminary agent (Navi) which researchers can adapt for their own needs. Navi is meant to be a capable starting point and template for the research/open-source community to develop and customize their own agents; together with our benchmark, our hope is that our contribution will enable faster reiteration in the research and development of AI agents.



**Benchmark cost**
- Our open-sourced repository README file provides a cost estimate for running the benchmark. This estimate includes both the VM cost (about 8 US dollars for all machines running in parallel for the duration of the evaluation), plus the models' costs (which can vary from 0 US dollars if the user runs a local VLM model, all the way to about 15 US dollars for GPT4o-mini or about 100 US dollars for GPT4o). 
- Appendix A.7 in our paper also includes a table of VM specs/costs and times. For times, we note that typical serial evaluation of a benchmark on the scale of Windows Agent Arena (or larger) would take significantly longer (likely multiple times longer) than our parallel setup.
- We note that these costs are similar to costs associated with evaluating state-of-the-art closed source models on other agentic benchmarks as well. We also note that as part of benchmark’s infrastructure, we do not need expensive high-performance compute as our parallelization can be done on inexpensive CPU cores (see **[N2]** in the "General Response to Common Questions from Reviewers") so any additional costs from running our benchmark beyond closed-model APIs are near-negligible. 

**Mistake in line 263**
- Thank you for highlighting this mistake. We meant to point the reader to appendix A.6 (TASK DEFINITION & CONFIGURATION). We will change this in our revised manuscript.

We hope our responses have addressed the questions that reviewers have. Please let us know if there are any other questions or suggestions.
We thank the reviewer for the valuable feedback. We are encouraged that the reviewer finds our work easy to deploy. We address each of your questions/concerns below:

**What are our contributions / How is this different from OSWorld?**
 - Please refer to **[N2]** in the "General Response to Common Questions from Reviewers" Official Comment at the top.

**Uniqueness of Windows tasks**
 - Even for web browsing tasks (which are not inherently OS-related), a significant amount of time in terms of engineering and testing was required to modifying the configuration and evaluation functions so that they would work natively on the Windows OS given differences across OS kernels/support. 
    - As just one example for web browsing tasks using Chrome and Edge, functions that access browser cookies, and settings in cached files are usually significantly different between Linux and Windows, and each task had to be modified and verified manually. We also added Edge browser support for several Web tasks, which did not exist in OS World. 
- Additionally approximately double the number of hours was spent creating new tasks for Windows-exclusive apps (Microsoft Paint, File Explorer, Settings, Notepad, Clock). These newly designed tasks are not only meant to work under Windows OS but also aligned with the actions and parts of workflows of common users of Windows---we will better clarify and describe the differences between adapted tasks, converted tasks, and newly designed tasks in Section 3 of our revised paper.

**Number of tasks**
- With 154 task templates, our benchmark falls in line with major benchmarks in the domain of execution-based evaluation such as MiniWoB++ (114 templates), AndroidWorld (116), Web Arena (241), and OSWorld (369). We are also actively monitoring the open-sourced repository and adding new tasks that we create/source ourselves as well as those made by the research community to expand the benchmark, including users who have already designed their own tasks within our benchmark.

**Agent design back-end**
- In Table 4 and Section 4, we describe the 32 different agent back-ends we tested along with different combinations of screen parsing and VLM reasoning models. These models range in size and other aspects from small local and cloud models (e.g. Phi3.5 and GPT4o-mini) to large cloud models (GPT4o and o1). We will better clarify and describe the design of our agent Navi in Section 4 of our revised manuscript along with the individual components and their specific considerations.

Please let us know if there are any other questions or suggestions and we'd be happy to answer!
We thank the reviewers for their time and greatly appreciate their insightful comments. We are particularly encouraged that all reviewers recognize the impact that a Windows-focused agent benchmark can have. We provide detailed responses to each reviewer’s questions below their individual reviews but also summarize our answers to the most common questions below:

======
- **[N1] Why only Windows?**
  - OSWorld is an agent benchmark majoritarily focused on agent tasks for non-Windows OS (~90% are Ubuntu/Linux tasks); Linux has ~4% market share among desktop operational systems. We focus this work exclusively on the Windows OS because, with 73% market share, it is the most widespread OS among PC users.
  - Despite its prevalence, Windows is still relatively under-represented in environments/benchmarks for agentic capabilities. Therefore, OS agents need to include Windows, and make use of its distinct UI, system-specific functions and command line, system-specific apps, and idiosyncrasies.
  - Given its uniqueness, proprietary nature, and under-representation as an OS, it is less clear how agents fare with even common actions/tasks in Windows. Differences between OS software ecosystems can significantly impact an agent’s observation/action spaces and performance, as noted by OSWorld itself (their agent performs ~2x better on Linux than Windows).
    - In our experiments, we see that common actions performed by humans in Windows can be challenging for state-of-the-art models that we use for our agents, suggesting that these tasks, the OS/environment, and the workflow aspects they embody still present specific challenges for agents.

======
- **[N2] What are our contributions / How is this different from OSWorld?**
  - In sum, the key differences and contributions are: **(1) OS focus (Linux vs Windows)** and **(2) parallelization of evaluation (time reduction from days to ~20-30 min, ~100x faster)**. 
    - **(1) OS focus**: We mention in our paper that ~2/3 of our tasks are adapted/inspired from OSWorld, and 1/3 are new. There is a non-trivial amount of engineering effort in porting tasks: for each individual task, we rewrite from scratch all configuration functions (which set the initial conditions for the task) and evaluation functions (which verify completeness) so they are compatible not only within a Windows OS but also within our custom parallelizable infrastructure. 
      - Section 3.2 understates this effort, which consumed ~800 person-hours for coding, testing, verification, etc. The remaining 1/3 new tasks consumed ~1600 person-hours. This excludes agent testing, etc.
      - **Only ~10% of OSWorld's tasks are Windows (43/412), We have >3x as many Windows-exclusive tasks.**
      - Also, despite not being an open-source OS, we made significant efforts to have Windows accessible via our benchmark as a self-contained docker image. We provide users a way to use a free evaluation copy which can be used for benchmark deployment and then easily/continuously renewed thereafter. 
      - As a result, our benchmark is one of the few (if not the only) that provides users open/free access to the Windows OS and computing environment for agent research and development. Our benchmark also allows users to install their own programs/applications, add and configure new tasks for their own needs, and more.

    - **(2) Quick/flexible evaluation via benchmark parallelization:** Agent evaluations based on task completion is inherently slow because of the sequential nature of tasks---especially as a benchmark grows in size and complexity. 
      - To our knowledge, we are the first to open-source infrastructure which allows researchers to fully parallelize evaluation on our benchmark in under 20-30 min for an arbitrary number of tasks, as opposed to tens of hours or even days or weeks of wait. This enables not only faster evaluation but also faster reiteration in research and development. 
      - With our benchmark, there is also *no* need for any high-performance computing setups or large storage capacities. Our setup relies on widely available and relatively low-spec inexpensive CPU cores and does not rely on any GPU resources (Table 10 in Appendix A.7 for details). Our setup also allows users to customize/configure the RAM allocated to the VM---allocated memory can be reduced down to as *small* as **2 GB** and still be able to run the environment. 
      - Users can install their own programs onto the image or VM and create their own tasks following details in our paper, README, etc.
      - We also note that our incurred costs are primarily from closed-source model API calls---which are similar to costs incurred from testing closed-source models on other agentic benchmarks as well. 

We hope our responses have addressed the common questions that reviewers have. Please let us know if there are any other questions. We thank the reviewers for their time!
General Response to Common Questions from Reviewers
Yes
No ethics review needed.
1. For all the designed tasks (Office, Web Browsing, etc), which is specific to the Windows environment? To distinguish WindowsAgentArena from OSWorld, authors need to discuss the unique tasks for Windows.

2. As stated within the paper, two thirds of the tasks are re-implemented from OS World and one third tasks are newly designed. It would be great if authors can show the newly designed tasks to help the audience clearly understand the contribution.

3. What is the backend model for the Agent? Have you tested on different backend models?
1. A new benchmark focuses on multi-modal tasks in the Windows environment.

2. Easy to deploy environment.

3. Authors design an extra multi-modal agent to validate benchmark’s effectiveness
The paper introduces WindowsAgentArena, a comprehensive benchmarking environment tailored for evaluating multi-modal agents in Windows OS settings. This platform builds upon the OSWorld framework, specifically adapted to provide 154 diverse, complex tasks representative of typical Windows applications and workflows, such as document editing, web browsing, coding, and system customization. The benchmark environment is designed for scalability, allowing for parallelized evaluation on Azure, significantly reducing the time required for agent testing. As a demonstration, the authors develop Navi, a multi-modal agent achieving a 19.5% success rate on WindowsAgentArena tasks, highlighting the challenges in reaching human-level performance (74.5%).
1. This benchmark does not show clear differences toward OSWorld

2. Agent design can be clearer

3. The number of tasks is limited
Yes
No ethics review needed.
（1）What’s the uniqueness of the tasks in the benchmark compared to OSWorld?
（2）Line 263 mentioned that the task procedure is shown in Appendix A.3, but the content in A.3 is about reward evaluation.
（3）Is it possible to extend the benchmark to other operating systems？If the OS difference matters to the agent evaluation, maybe this is necessary
（1）The paper proposes a new benchmark specifically targeting LLM agents’ performance in complex, real-world tasks on Windows OS. The benchmark is well-designed, with a diverse set of tasks that test multiple aspects of agent capabilities. 
（2）The paper provides adequate evaluation on the proposed benchmark with a proposed baseline agent and validates its competitive ability.
This paper introduces WindowsAgentArena, a benchmark designed to evaluate LLM agents in performing multimodal tasks within the Windows OS environment, including 154 tasks covering office software, web browsing, coding, etc. The benchmark assesses the ability of LLM agents to interact with a complex operating system through various modalities. The paper also provides a baseline agent Navi and test its performance on the benchmark. Codes and deployment guidance are open-source.
（1）Although the benchmark is claimed to be extensive, it is specifically focused on Windows OS. I’m not sure whether this benchmark could be adapted or extended to other operating systems or environments, as this could broaden its applicability.
（2）Apart from the difference in operating environments, it is unclear how the tasks in this benchmark uniquely assess agent capabilities compared to OSWorld. It seems that the tasks have much overlapping with those in OSWorld. More discussion about key differences is needed to distinguish this work from existing works.
（3）I’m concerned about the usage cost of the benchmark. An API cost estimation for running the tasks in the benchmark is needed.
Yes
No ethics review needed.
See weaknesses.
1. WindowsAgentArena fills a gap in agent benchmarking by focusing on the Windows OS, which is widely used but less explored in agentic evaluations.
2. The ability to parallelize the evaluation on Azure for faster task completion enhances the framework's practicality and adaptability for large-scale studies.
3. By including tasks across multiple categories (e.g., document editing, web browsing, system management), WindowsAgentArena effectively simulates a real-world computer environment.
4. Detailed performance metrics, examples of successful and failed cases, and analyses of task complexity offer clear insights into the capabilities and limitations of agents.
This paper presents WindowsAgentArena, a benchmarking suite designed to evaluate multi-modal agents' abilities within a Windows OS environment. Building on previous work like OSWorld, WindowsAgentArena focuses exclusively on the Windows operating system, making it possible for agents to perform diverse tasks representative of real-world computer usage, such as document editing, web browsing, and multimedia manipulation. The study introduces a novel agent, Navi, which demonstrates the framework’s capabilities, achieving notable results in task performance compared to humans. The benchmark suite also emphasizes scalable and fast evaluation through parallelization on Azure, which reduces evaluation time significantly.
1. Windows OS is proprietary, which may limit access for researchers who rely on open-source environments or who cannot accommodate the licensing and resource costs associated with a Windows-based setup.
2. Running Windows virtual machines (VMs) with multiple agents and parallelized tasks can be resource-intensive. This may necessitate high-performance computing setups and large storage capacities, particularly for cloud-based evaluations. Configuring and maintaining the Windows environment, including installing software, handling updates, and ensuring compatibility with the benchmark suite, can also be complex and time-consuming.
3. Agents trained and evaluated solely in a Windows environment may develop strategies tailored to Windows-specific UI elements and workflows, potentially limiting their ability to generalize to other OS environments, computing platforms, or even to updated versions of Windows.
4. WindowsAgentArena builds upon OSWorld’s approach but is specific to the Windows environment. This focus on a different OS may not represent a fundamentally novel leap, as it largely adapts pre-existing methodologies rather than introducing groundbreaking new concepts.
5. Many tasks in WindowsAgentArena replicate basic or common actions found in Windows OS, such as file management or web browsing. The tasks may lack complexity or novelty that could reveal new challenges for agentic behavior, making the approach feel less groundbreaking.
6. The Navi framework leverages established multi-modal approaches, such as UIA tree parsing, DOM tree extraction, and OCR for screen parsing. While effective, these methods are widely used in similar environments, potentially limiting the originality of the agent interaction techniques.
7. The reward structure is based primarily on task completion, similar to other agent benchmarks. WindowsAgentArena does not seem to innovate with intermediate or adaptive reward mechanisms that might encourage more nuanced agent learning behaviors.
8. The framework relies on traditional Set-of-Marks (SoM) and bounding boxes for visual grounding, which are widely used in agent evaluation. Introducing innovative ways to handle visual-language misalignment might have added novelty to the approach.
Yes
No ethics review needed.
Is Navi just designed as a baseline to this benchmark or an innovative agent method? 
Is there a plan to test on more benchmarks for evaluate the performance of Navi in the additional results?
The writing of this paper is well. The problem definition is very detailed and easy for the reader to understand. The benchmark is a good complementary to the existing work and the claimed evaluation system is more efficient to the current benchmark system.
This paper introduced a OS Agent benchmark on Windows OS, and proposed a new MLLM agent called Navi. The goal of this benchmark is to solve the challenges of  restricted modalities and slow process in agent evaluations. The benchmark defined 150+ create 150+ diverse Windows tasks across different domains. Navi is designed with chain-of-thought prompting and tested on this benchmark with different SoMs and MIND2WEB Dataset.
The experimental results lack the comparison of well-known agents, but just use different settings to test on multiple base models. The design of Navi agent is relatively simple and motivation is not clear. The additional experiments on MIND2WEB is not enough to support the agent performance.