Large Language Models Can Self-Improve At Web Agent Tasks
llm;llms;synthetic data;web agents;agents;self-improvement;unsupervised learning
Yes
No ethics review needed.
- What is the definition of plausible trajectories and high-quality?
- What are the possible oblations against the proposed type of data?
- well written.
The papers is about having an agent for environments with lack of training data like web browsing. In that way, LLM can be a good candidate. In this work, they introduced a new technique in which LLMs can self-improve their performance as agents in long-horizon tasks in a complex environment using the WebArena benchmark. To better understand the effect of the self-improvement method, they introduced two auxiliary metrics: 1) a measure to analyze capabilities acquired and lost by the agent and 2) an extension of the VERTEX score to measure the quality of variable-length agent trajectories.
- The innovation of the work is very limited! 
- Lack of enough experiments! The experiments are not comprehensive! It should consider a good analysis across different datasets and careful reasoning of what is going there! For example, having complete oblations studies versus other possibilities.
- Lack of comparison with baselines! There are a lot of works of self-improvement and self-correctness! There should be considered as baselines. 
- This work needs a proper and further study and analysis! This is an incomplete work to be published!
Yes
No ethics review needed.
While the study is promising, its impact would be further enhanced by addressing the concerns regarding the selection of certain variables used in the experiments and evals and model diversity. 

### Suggestions for Improvement:
- **Expand Hyperparameter Exploration:** Providing a sensitivity analysis or a justification for the selection of hyperparameters used in generating synthetic data would enhance the transparency and reproducibility of the results.
- **Broader Agent Evaluation:** Including diverse model architectures in the experiments could help validate the proposed self-improvement techniques across a range of LLMs, potentially increasing the impact and applicability of the findings.

I would be happy to increase the score post author responses.
### strenghts:
1. **Capabilities through Self-Improvement:** The paper demos how LLMs can extend their capabilities through self-improvement techniques, particularly in the context of complex, long-horizon web agent tasks. This ability to acquire new capabilities while largely retaining existing ones is notable.
2. **Appropriate Eval Metrics:** The introduction of novel metrics, and scores to evaluate the quality of trajectories, adds depth to the evaluation process. These metrics provide a nuanced view of the models' performance, moving beyond simple task completion metrics and enabling a more detailed assessment of the agent's robustness and capabilities. the additions seem appropriate  enough to capture both success, relevance, and efficiency of trajectoriesl
3. **Synthetic Data Utilization:** The use of both in-domain and out-of-domain synthetic data for fine-tuning is commendable. This approach not only addresses the challenge of data scarcity but also demonstrates a systematic mechanism for enhancing agent model generalization.
solving complex, multi-step, long-horizon tasks using prompting or fine-tuning poses various challenges. The paper presents a sound approach to enhance the capabilities of LLMs through self-improvement on web automation task. They propose evaluation techniques and effective use of in-distribution and out-of-distribution synthetic data, showcasing some performance gains.
Good to address:
1. **Hyperparameter Selection:** The paper lacks a clear justification for the choice of hyperparameters in the synthetic data generation process and generating new objectives. e.g. using 4 or 2 few-shot samples, temperature value, and perhaps just one sentence on why 0.7 cosine similarity used is better. Any missing details may lead readers to question the replicability and robustness of the results. A more thorough analysis or rationale for these choices would strengthen the paper.
2. **Limited Model / Results:** The study's reliance on a single or fewer models limits its generalizability. Expanding the experiments to include a subset of models on appropriate agent frameworks that came out recently, such as smaller or different architectures, could provide insights into how these techniques scale or vary across different model sizes and types. I am thinking aloud if the results section can be tightened and elaborated more. But I will wait to see if my other reviewers has some feedback or ideas on this front.
Yes
No ethics review needed.
How many examples are in Mixture C? How does the result scale with the number of examples used in Mixture C? Does it improve as more examples are provided?
- The paper shows that you can use in-domain trajectories as few-shot examples to generate novel out-of-domain trajectories, which will be useful in producing synthetic data for finetuning agents.
- The paper proposes a new metric $\text{VERTEX}_{\text{DTW}}$ which measures the similarity of an agent’s trajectory w.r.t. a reference (in this paper, the reference is a gpt-4 trajectory). This can be useful for providing a soft metric that has less false negatives than the WebArena functional correctness evaluation.
This paper proposes a method of improving LLMs for web agent tasks. The method involves prompting an existing LLM as an agent to tackle tasks in WebArena. The trajectories are then filtered down to retain those that are likely to be successful. The model is then finetuned (with QLoRA) on these trajectories. In the paper, alternative trajectory generation methods are also explored, such as using the successful WebArena trajectories to generate out-of-domain trajectories on other websites. In addition, the authors propose a new metric for measuring the similarity of an agent trajectory, which has benefits over using the functional correctness evaluation in WebArena.
My primary concern with this paper is that its evaluation setup is not fair. The models are essentially *training on the test set*, and doing a pass@2 on the evaluation set (these results align with the ablation experiments in the paper, where training on Mixture C actually makes the agent worse). Consider the filtered subset of in-domain examples (mixture A). It has a very high accuracy rate (0.919), which is expected as tasks that errored out or the model failed to solve are filtered out. In the WebArena benchmark, models often only stop when they are successful, so the false negative rate is low. If you view mixture A as groundtruth samples, finetuning on this subset is essentially allowing the model to memorize the examples that have passed, while ignoring those that it failed on the first attempt. When you test such a model, this can be viewed as essentially taking a second sample for the failed tasks. This is not a fair comparison to baseline models: a fairer comparison would be the same baseline model but an accuracy score of its pass@2 (i.e., take the best run out of two samples using the same filtering criteria). [1] actually does this with a prompted value function and gets a similar relative improvement (29%) compared to this work, and they do not do any training on the samples. If the authors can (1) compare their results against pass@2 of the agent (while resetting the WebArena environment after the first run, to avoid bias), and (2) perhaps run their model on a different benchmark, e.g., Mind2Web, VisualWebArena, WorkArena, or WebLINX, to show generalization, this would alleviate some of my concerns.

In addition, it would also be good to have error bars on the evaluation results as WebArena can be noisy, as the absolute difference between the baseline and the self-improved agent is low (7.14 to 9.36) and could be within the error range.

The proposed $\text{VERTEX}_{\text{DTW}}$ metric is evaluated against gpt-4 generated outputs, which possibly introduces model bias in the evaluation. WebArena has [human demonstrations](https://github.com/web-arena-x/webarena/blob/main/resources/README.md#12212023-human-trajectories), and it would be good to compare against these instead.


Overall I think the evaluation of the paper can be greatly improved (by measuring on more benchmarks, to avoid train-test contamination), as well as more ablation experiments. For these reasons I don’t think that the paper in its current shape is sufficiently complete.
Yes
N/A
No ethics review needed.
There are a few other points that I would appreciate if the authors could clarify.

**(1)** The authors claim that the procedure for filtering in-domain self-refinement data is entirely unsupervised, but then seem to contradict this point later on.

In Lines 135-136: “Similar to the self-improvement prior work we discuss earlier, the collection of this data is completely unsupervised and no ground-truth labels are utilized for filtering and selection.”

Then, in Lines 194-196: “Finally, we also filter out any trajectories where the WebArena environment encountered an error or the model failed to produce a valid, parsable generation”
    
If I understand the procedure correctly, the highlighted parts of the filtering process are not truly unsupervised — they leverage signals that are external to the model, e.g. feedback from the environment. If so, the first claim should be adjusted accordingly.
    
**(2)** On lines 290-292, the authors write: “We note, however, that a number of tasks in the WebArena benchmark are trivial tasks and can be solved by a trivial baseline agent or weak model that performs no actions and only immediately exits by always generating stop [N/A].” 

This seems like a major weakness of the benchmark. What proportion of the WebArena tasks have this degeneracy property? How many of the <10% of tasks solved by tested LLM agents are degenerate? From my understanding of the text, degenerate tasks **are** filtered from capability score computation, but **are not** filtered from/identified during evaluation. Is this correct?
- The study of generalizable techniques for improving LLM agents is of interest to the community. While many previous works have studied the advantages and limitations of self-refinement for reasoning and simple decision-making tasks, there has been limited work on harder benchmarks with longer decision-making horizons, like Web Arena. These tasks represent the frontier of state-of-the-art LLM capabilities. As a result, developing more effective methods for self-refinement on such problems is especially important.
- The paper is polished, clear, and fairly well written. The authors expound on all experimental details.
- The default metric for LLM performance on WebArena is the average completion rate of environment tasks. The authors make an effort to address the lack of granularity in this metric by introducing two auxiliary metrics for assessing LLM agentic performance. These auxiliary metrics add some nuance to the analysis, though this could be strengthened by increasing the number of experiment replicates and/or increasing the inference-time sample size, as models are evaluated at high sampling temperature.
The authors study whether self-refinement (i.e. finetuning on self-generated data) can improve the performance of large language models on web agent tasks via the WebArena benchmark. This benchmark tests agents on a set of web page decision-making tasks. In such setting, self-refinement data consists of trajectories of interleaved environment observations and LM agent actions. The authors test three different ways of generating  trajectories for self-refinement: (A) fully “in-domain,” using train set tasks from the Web Arena environment only; (B) on a “in-domain” and “out-of-domain”  mixture, using train set tasks from WebArena as well as synthetic tasks generated by the LM agent; (C) fully “out-of-domain,” using only synthetic tasks. All “in-domain” trajectories are filtered using self-critique, for LM agent generation errors, and environment errors. The authors finetune a single LLM on each of these three data mixtures and compare the resultant LLM agents’ performance on WebArena under three different metrics. They show that self-refining on data mixtures A and B result in some improvements to performance on 3/3 metrics, while finetuning on mixture C produces a slight drop in performance on 2/3 metrics. Finally, the authors show that running an additional round of self-refinement with the best performing data generation strategy (mixture A) causes a drop in performance on 3/3 metrics, compared to results after a single round.
While the questions posed by this paper are timely, the experimental results have limited scope and are much more mixed than the text would suggest. While the title reads “LLMs can self-improve at Web Agent tasks,” the results seem to suggest that the performance margins are extremely limited. 

- Throughout the paper, the authors cite a 31% improvement over the base LLM agent as evidence for the power of self-improvement. However, this 31% is a relative score, corresponding to a mere ~2.2%  improvement in raw score from a completion rate of 7.14/100 to 9.36/100. The statistical significance of this improvement is unclear for two reasons: (1) the scale of standard error over 0-1 task completion score is omitted from Table 3; (2) according to Table 5, the authors sample from models during inference with a high temperature (T = 1). Sampling rather than using greedy decoding introduces an additional source of statistical error to results, which is similarly not quantified.
- In the introduction, the authors write: “We analyze three synthetic training data mixtures and find all three mixtures improve performance” (Lines 101-102).  This claim is misleading, since as shown in Table 3, self-refining on the third data mixture (C, “out-of-domain” tasks only) produces decreases in performance on 2/3 of the studied metrics.
- The authors show that iterative self-refinement produces a drop on all 3/3 Web Arena LLM agent performance metrics, even after the careful filtering procedures (Table 3, last row). As a reader, this result powerfully contradicts the tone of the title.
- On lines 404-406, the authors write: “Interestingly, we find that the majority of capabilities acquired by M_A and M_C are mutually exclusive, suggesting in-domain synthetic examples and out-of-domain synthetic examples improve acquisition of different capabilities.” This point would be extremely interesting if strengthened, but does not seem to be rigorously supported by the results. Currently, it seems to be being made on the basis of a single random seed of self-refinement in each setting, i.e. using a single sampled set of high temperature synthetic task generation + agent trajectories + LLM fine-tuning. In my view, to confirm these results, evaluation should be conducted with at minimum ~3-5 seeds in each data mixture setting.
- Overall the experiments have limited scope. Evaluations are conducted with a single seed for each data generation variant and a single LLM.  

Though the results of this paper are mostly negative, I still believe they are of interest to the community. I would be willing to increase my score if (1) the authors can provide an estimate of statistical error for all reported scores; and (2) the tone of the paper is adjusted to reflect the strength and scope of the experimental results.
In this work, we fine-tune large language models on synthetic data to self-improve at web agent tasks and evaluate on the WebArena benchmark, achieving a 31% improvement over the base model through our self-improvement procedure.
@inproceedings{
anonymous2024large,
title={Large Language Models Can Self-Improve At Web Agent Tasks},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=jwME4SY0an},
note={under review}
}
Training models to act as agents that can effectively navigate and perform actions in a complex environment, such as a web browser, has typically been challenging due to lack of training data. Large language models (LLMs) have recently demonstrated some capability to navigate novel environments as agents in a zero-shot or few-shot fashion, purely guided by natural language instructions as prompts. Recent research has also demonstrated LLMs have the capability to exceed their base performance through self-improvement, i.e. fine-tuning on data generated by the model itself. In this work, we explore the extent to which LLMs can self-improve their performance as agents in long-horizon tasks in a complex environment using the WebArena benchmark. In WebArena, an agent must autonomously navigate and perform actions on web pages to achieve a specified objective. We explore fine-tuning on three distinct synthetic training data mixtures and achieve a 31% improvement in task completion rate over the base model on the WebArena benchmark through a self-improvement procedure. We additionally contribute novel evaluation metrics for assessing the performance, robustness, capabilities, and quality of trajectories of our fine-tuned agent models to a greater degree than simple, aggregate-level benchmark scores currently used to measure self-improvement.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
llm llms synthetic data web agents agents self-improvement unsupervised learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/a102bb045e595e2329d931d532f58bfcd696e2de.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/206f86832242309efcba12c73509c26ce9b74604.zip
Large Language Models Can Self-Improve At Web Agent Tasks
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission