HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scale
Multi-Agent;Large Language Model;Software Engineering
Yes
No ethics review needed.
The result and analysis shows the developed system works well, how ever it is not clear how generalizability and scalability is presented in the paper. It is recommend that authors clearly and explicitly explain how you achieved generality, efficiency, and scalability, what experiments you conducted, and how you evaluated the result section. Or/and add  section on generalizability and scalability experiments, because as for "scalability", there are 3.2 AGENT COMMUNICATION AND SCALABILITY, but ther eare no such description on other 2 features.
It showed the avility to solve multipole software engineering tasks such as issue resolution, code generation, fault localizaiton and repair. The design enable users to use lightweight (cost efficient) LLM for navigation agent and strong LLM for other agents, depending on the problem.
This paper presents LLM based multi-agent system for various SE tasks, consisting of Navigator, Editor, Executor and Planner to control there 3 agent modules.
The paper shows the versatility of HyperAgent by applying 3 tasks, i.e. issue resolution, code generation, fault localization and repair, along with comparison with SOTA models, and how it outperforms other methods. The authors also provide a detailed analysis of agent roles, tool design and behaviour,
Implimantation details is not clear until the appendix. For example, Footenote 1 on  page 2 says" details of each agent are writtin in Sec4". But  Sec.4 is too short to understand. This is not "detail", at least "Implimentation summary" or "overview". Please add  more details on the architecture of each agent, their interactions, or the key algorithms/prompt used, for example.

How much the system is scalable and how did it implimented is not clear. From sec 3.2 denoted it is scalable because of using message queue. But it is not clear how it is related to scalablity. I understand that MessageQueue is designed for large system integration, but not show how it is related to "numerous subtasks, handlig complix tasks efficiently", which the authors wrote on Introduction.
Yes
No ethics review needed.
1. Is the Planner sequential or can it also dispatch parallel queries to multiple agents.
The paper provides a clear framework for decomposing software engineering tasks and solving these in a specialised manner. Specifically, the tools proposed by the paper, as well as how they are called (writing python code) are useful contributions. The ablations show tool design adds significant value to the framework, particularly the choice of what controls are given to the LLM (e.g. code_search, keyword summaries, and repair editor). The paper provides broad evaluation on both generation tasks (RepoExec) as well as bug-fixing (Defects4J and SWE-bench) which support claims of generalisability.
The paper proposes an LLM agent approach for software engineering tasks. It identifies that SE tasks often involve a combination of localisation, editing and execution, all orchestrated by some higher-level plan. A framework made up of modular components for each of these is proposed where each component is a ReAct style LLM agent loop with access to specialised tools. Through positive results on software generation (RepoExec) and bug-fixing (SWE-bench and Defects4J), the paper aims to show generalisability and efficiency (in terms of API costs).
1. The paper lacks thorough comparison to prior approaches that employ similar multi-stage approaches. CodeR [1] uses a multiagent approach for with seperate manager, localiser, editor and verifier (very similar to the components of Hyperagent), achieving 28.3% on SWE-bench-lite. MASAI [2] uses a modular architecture with similar components (without a manager) achieving 28.3% on SWE-bench-lite. Approaches such as AutoCodeRover (v20240620) (30%) [3] and Aider (26.3%) [3] both have phases for localisation and editing. Overall, the novelty of such a multi-stage approach is questionable, especially since the performance on SWE-bench is comparable to similar approaches. 

2. The paper also does not provide sufficient evidence as to why a multi-stage approach in general is more "generalisable". The performance numbers quoted for agentless (24.3%) seem out of date, with Agentless + GPT 4o (2024-05-13) achieving 27.3%. While it is understandable that comparing with all new entries on swebench is unreasonable, comparing with one of the many strong non multi-stage approaches both quantitatively and qualitatively would do much to strengthen the papers claims. 

In general, while the paper demonstrates positive results of their approach on two different kinds of tasks, the novelty of the approach as well as comparisons with prior work are lacking.

[1] https://arxiv.org/pdf/2406.01304
[2] https://arxiv.org/pdf/2406.11638
[3] https://www.swebench.com/
Yes
None
No ethics review needed.
- Line 42: “their claim of addressing general software engineering tasks is overstated”. Why is this the case? Given that the downstream tasks you evaluate on are all evaluated on Python or Java, how does HyperAgent resolve the shortcomings of more versatility across tasks, languages, and dev. Scenarios?
- Lines 45 - 104: You introduce a workflow and claim that this is representative of most of software engineering - what evidence is there to reflect this? This seems very reminiscent of the issue resolution pipeline that prior works already introduce. Also, “Although different SE tasks require varied approaches, they all follow a similar workflow.” - Are there citations or prior evidence from existing SE works that reflect that the workflow you discuss is in fact the most reflective? This part of the introduction feels quite anecdotal.
- Line 99 - How are your three advantages made clear in your evaluation? Aside from higher performance, I do not see why existing works are less generalizable than HyperAgent, or why these agents are more efficient or scalable.
- Line 116 - What is SWE-bench-Python, RepoExec-Python, and Defects4J-Java? Are these variants of the original benchmark?
- Line 118 - SWE-agent and OpenDevin were both evaluated on non SWE-bench tasks, including HumanEval, HumanEvalFix, Bird, BioCoder, WebArena, and much more. The claim that this is the first system to work on many SE tasks feels like an over-claim.
- Line 254 - “A task can *be* defined…”
- Line 251 - Missing space between “HyperAgent” and “is”
- Line 328 - What does processing time account for? Does this include or exclude the calls to the API? What’s the variance on these numbers?
The HyperAgent’s ability to generalize to multiple benchmarks is its main strong suite in my opinion. The adaptation of existing multi-agent methods to a larger span of software engineering tasks is great to see, and it is a good confirmation of the methodologies suggested by prior works. This work explicitly identifies clear stages of software engineering, then demonstrates how a multi-agent framework designed around these stages is extensible to many tasks. In addition to the implementation, HyperAgent demonstrates comparable performance to existing solutions across three benchmarks.
This work introduces HyperAgent, a multi-agent software engineering system that leverages Language Models. The HyperAgent system can be adapted to work on multiple Software Engineering benchmarks, specifically SWE-bench, RepoExec, and Defects4J. HyperAgent consists of one “parent” agent (the Planner) that can invoke one of three “child” agents (Navigator, Code Editor, Executor). These child agents can be invoked in parallel in any order by the Planner. The implementation also includes a suite of tools tailored to each child agent, along with flexible problem templating that allows for easy adaptation of HyperAgent to different coding tasks. HyperAgent performs comparably on several benchmarks, and the authors put forth several ablations examining how HyperAgent performs on such tasks.
While the performance improvement of HyperAgent is admirable and the task coverage is quite impressive, I think it’s not quite clear what distinguishes HyperAGent from prior work. The methodology feels like a slight adaptation of existing multi-agent systems (e.g., MetaGPT) for the SWE-bench benchmark. The ablations, while interesting, look very similar to the ablations presented in SWE-agent. It is great that HyperAgent demonstrates comparable performance on several software engineering and coding benchmarks, and while this feels like a good engineering contribution, it remains unclear to me what new research questions or conclusions this exploration can offer up. I believe that more discussion about how HyperAgent’s architecture and methodology differs from prior work, along with analyses that back up these conclusions would make this paper a stronger research contribution.

- Section 2 Related Work: I would’ve liked more investment into a discussion of how HyperAgent differs from prior works, particularly in the second half of Section 2.2. After reading Sections 1 and 2, it is still not clear to me what new research questions HyperAgent addresses or what it does that truly distinguishes it from prior work.
- Section 3: Several comments - (1) How is this approach different from existing work such as MetaGPT, which arguably has more complex multi-agent systems following the Standard Operating Procedure framework? (2) Why is Multi-agent better than existing single agent approaches? How does having multiple agents lower inference costs or eliminate redundant information (wouldn’t the same information have to be communicated to multiple agents several times?) Also, it sounds like multiple child agents can be spawned in parallel - how is this more efficient than single agents, and why is the performance better because of this?
= The engineering details discussed in Section 3 sound quite impressive, particularly the use of the Message Queue and the general coordination of multiple agents. However, it is left to the reader to determine why these tools or multi-agent systems are interesting or novel relative to prior work. For instance, for Section 3.3, it is cool that these tools are made available, but unclear why this is better than the search and localization tools that prior works introduce (e.g. edit, search_dir, search_file in SWE-agent).
- Section 6 - These analyses are almost identical to the ones in SWE-agent. While this is interesting information, the paper may be stronger if there were more novel examinations that demonstrate how HyperAgent is distinct from prior work. For instance, “6.2 Analysis of Tool Design” is very similar to Table 3 in the SWE-agent paper. “6.3 Agent Behavior” is the same kind of graph as SWE-agent Figure 7. Analysis 6.1 is interesting, but it would be nice to know why the performance drops are not equal, and how certain tasks are still resolved even when there is no Editor, Navigator, or Executor.
Yes
No ethics review needed.
- Can the authors compare HyperAgent to Agentless (on SWEBench-Verified) and ChatRepair (on Defects4J)? Specifically, could you include average runtimes for Agentless and SWEAgent on SWEBench?
- For both RepoExec and Defects4J, can the authors include the results for at least one of the Full configurations? This will help with understanding the scalability of the approach to better models.

Minor clarifications which I believe could enhance the readability of the paper:

> For example, multiple Navigator instances can explore different parts of a large codebase in parallel, the Editor can apply changes across multiple files simultaneously, and the Executor can run tests concurrently, accelerating validation.

- Can the authors clarify if there are specific design decisions made to ensure synchronization between agents preventing them from modifying the same files?

> Failed tasks are requeued for reliability
> HYPERAGENT also has a problem of early exit (due to hallucination that the task has been
solved)

- As pointed out by the authors, it could be challenging for the LLMs to determine success/failure of task/subtasks. Can you clarify how are failures detected for requeuing?

- What are the details of HyperAgent-lite-3?

- Explored the sharing of previous context with a specific agent?

- Can the authors clarify if they intend to make the HyperAgent source code open source?

- Can the authors include a discussion about LSPToolKit and if there are specific design choices that were made to better align it as a tool for LLMs? Specifically, can the authors compare with some of the existing approaches that use LSP to build tools for LLMs. This could be instructive for devtool builders.
The paper demonstrates a generalized software engineering agent. As such, generalizability of the approach, and strong demonstration (even if not state-of-the-art on individual benchmarks) across 3 different tasks is the main strength of the paper. The proposed multi-agent breakdown is simple, while achieving good performance, with ablations supporting the need for each component as well. 

The paper also demonstrates the usefulness of multi-LLM multi-agent system through the HyperAgent-Lite configurations.

The application of dynamic message queues for asynchronous communication between the planner and child agents allows parallelizability and reduction in runtimes.

Further, the authors also identify the difficulty faced by LLM agents in using various IDE provided tools (like go_to_definition, which for example, actually reduced the resolve rate), and came up with augmentations to improve the tool (w/ search ablation in go_to_definition, etc.), which is instructive. This is particularly insightful, and points to further research in the direction of building better abstractions around existing code-tools that will be more suitable for LLMs.

The paper is written clearly, and easy-to-understand.
The paper introduces HyperAgent, a multi-agent system consisting of 4 primary agents (Planner, Navigator, Code Editor and Executor) organised in a Star-topology, to achieve generalizable software engineering capabilities across languages (Java and Python) and tasks (Issue-PR resolution over SWE-Bench, Bug localization and Repair over Defects4J, and repository-level code generation over RepoExec.). The planner agent creates subtasks and issues messages over async queues to each of the other 3 agents, which then try to execute the subtask. An LLM summarizer condenses the child agent trajectory to be sent back to the planner.

The different agents in the system can be instantiated with different LLMs, providing the ability to achieve a cost/capability tradeoff depending on the difficulty of the task.

The authors present results on 3 different software engineering tasks (end-to-end issue resolution, bug localization, bug repair, code generation) to demonstrate the generalizability of the proposed system.
I have two primary concerns with the paper:
- *Effectiveness of multi-agent setup*: The proposed system drives the complete workflow with an LLM agent (the planner), which is in direct contrast to the proposal by Agentless, which achieves better results than the best HyperAgent configuration on SWEBench-Verified, while also being much more cost effective (\\$0.34 for Agentless vs \\$1.82 for HyperAgent). This calls into question whether a comparatively complex agentic system is required for the software engineering task.
- *Evaluation baseline for Defects4J* - I would suggest the authors compare HyperAgent to ChatRepair [1], which reports 114 solved cases on D4J v1.2. I seek to understand how HyperAgent compares to ChatRepair and if the proposed multi-agent approach offers benefits above the ChatRepair. 

[1] Xia, C. S., & Zhang, L. (2023). Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT. arXiv preprint arXiv:2304.00385.
Yes
No ethics review needed.
1. For the executor, how does it find appropriate unit tests to execute? If defect fixing requires introducing unit tests that do not currently exist in the project, how does this agent function in such scenarios? This is a major shortcoming of the paper, and it lacks a detailed analysis in this area.
2. In Section 5.1, how does HypeAgent perform compared to other models on SWE-agent?
3. In Section 5.3, how do other open-source agents in Section 5.1 perform on this dataset?
1. This paper designs a multi-agent system with four components: an agent for planning implementation, an agent for retrieving file content within the project, an agent for making code modifications, and an agent for executing unit tests.
2. The overall process control of these agents is managed by a planner, and corresponding APIs are defined for each agent to interact with the project.
3. This paper conducts relatively solid experiments on three datasets.
This paper proposes a multi-agent framework to address general issues in software engineering development. The multi-agent system is designed with four components: an agent for planning implementation, an agent for retrieving file content within the project, an agent for making code modifications, and an agent for executing unit tests. The overall process control of these agents is managed by a planner, and corresponding APIs are defined for each agent to interact with the project. Experiments were conducted on three datasets: SWE-bench, RepoExec, and Defects4J. However, the final experimental results did not demonstrate that the proposed method is significantly stronger than other baseline methods.
1. Compared to previous work, such as ACR and SWE-Agent, the innovation in this paper is primarily the addition of an Execution Agent component; the other agents are almost identical to those in previous work.
2. The HypeAgent proposed in this paper does not show significant improvements over previous methods in the final experimental results.
3. The HypeAgent proposed in this paper does not compare its fault localization performance against SWE-Agent.
Yes
No ethics review needed.
Please see weakness section above.
* The paper focusses on an important problem of developing generalist agents without being tailored for specific SWE or coding tasks (e.g., competition coding etc)
* The combination of subagents with asynchronous message queues for improving subtask efficiency and parallel execution (e.g., parallel editing across multiple files) is also interesting and novel
* Quantitative evaluations across multiple benchmarks are provided to demonstrate the generalizability across different coding tasks and programming languages.
The paper focusses on the recently increasingly popular area of developing autonomous software agents for coding tasks. In particular, the paper focusses on the development of a software agent for generalizability across different tasks (github issue, repair etc) and languages (python, java). To this end, the paper proposes Hyper-Agent which uses four specialized agents - planner, navigator, code editor and finally code executor for scaling to generic SWE tasks. The planner agent acts as the main agent and is responsible to collaborating with multiple specialized agent in an asynchronous manner, in order to perform the final task. The specialized agents improve subtask execution performance while the asynchronous queue based execution improves throughput efficiency. Quantitative evaluations for three SWE benchmarks are provided in order to prove the efficacy of the proposed approach.
While the key idea and problem is interesting, I have some questions regarding some experimental results and details,
* The idea of using the same base-LLM to act as different subagents has been previously explored before [1,2]
* How are the subagents specialized? Is the difference primarily form using different input contexts / system prompts, tools etc.
* The paper claims that the final agent is generalist and applicable across diverse tasks. However, its not clear if the context and implementation for each task is specialized or not? 
      - For instance, when solving the swebench like tasks, the original swe-agent implementation provides task specific instructions / tips (e.g., reproducing the issue) in the input context. Is a similar strategy also applied for  hyper-agent?
* Also regarding the experimental evaluations, are the baselines and the proposed approach using the same base-llm (e.g, Tab. 1)?
    - In particular, according to Tab. 1, 7: the best performing hyperagent configuration using claude-3-sonnet, however some of the baseline results are reported with gpt4o.
* Also the numbers with some of the baselines seem off: autocoderover with gpt4o performs 30.67% on swebench-lite and 38.40% on swebench-verified (both higher than hyperagent). However, the reported numbers for the same in Tab. 1 seem to be much lower.
* Finally, while a minor point, I had some concerns regarding the usability in terms of avg. cost which is for instance 1.82 on swebench for the best hyperagent configuration. In contrast, a simple 3 stage model (localization, repair, rerank) with agentless achieves $0.34 per instance on average while achieving better solve-rate. 
* How are the ablation studies performed in Sec. 6? For instance, w/o navigator which agent is used for code navigation the planner agent or any other subagent.
* Are there some incontext examples teaching which subagent should be called by the planner agent? If not, is it mostly reliant on the functon calling capability of the main planner agent?

References:

[1] Meta{GPT}: Meta Programming for A Multi-Agent Collaborative Framework, Hong et al, 2024

[2] OpenDevin: An Open Platform for AI Software Developers as Generalist Agents, Wang et al, 2024
@inproceedings{
anonymous2024hyperagent,
title={HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scale},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=PZf4RsPMBG},
note={under review}
}
Large Language Models (LLMs) have revolutionized software engineering (SE), demonstrating remarkable capabilities in various coding tasks. While recent efforts have produced autonomous software
agents based on LLMs for end-to-end development tasks, these systems are typically designed for
specific SE tasks. We introduce HyperAgent , a novel generalist multi-agent system designed to
address a wide spectrum of SE tasks across different programming languages by mimicking human
developers’ workflows. Comprising four specialized agents—Planner, Navigator, Code Editor, and Executor—HyperAgent manages the full lifecycle of SE tasks, from initial conception to final verification.
Through extensive evaluations, HyperAgent achieves state-of-the-art performance across diverse
SE tasks: it attains a 26.00% success rate on SWE-Bench-Lite and 33.00% on SWE-Bench-Verified for
GitHub issue resolution, surpassing existing methods. Furthermore, HyperAgent demonstrates
superior performance in code generation at repository scale (RepoExec), and in fault localization and
program repair (Defects4J), often outperforming specialized systems. This work represents a significant
advancement towards versatile, autonomous agents capable of handling complex, multi-step SE tasks
across various domains and languages, potentially transforming AI-assisted software development
practices.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Multi-Agent Large Language Model Software Engineering
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/5d63247f0c73200487f9f7bc356d0989d7a04b3f.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/93d460b7cbed5a4629b67b85b34abc9566c0e058.zip
HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scale
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission