ReFeR: Improving Evaluation and Reasoning through Hierarchy of Models
Multi-agent framework;reasoning;evaluation;multimodal;hierarchy;LLMs;VLMs
Yes
No ethics review needed.
1. The paper claims that on line 51-52 the framework promotes explainability and robustness.
 - Explainabiltiy: It is not clear how this happens? Why is trying to use another LLM to explain another LLM's output sound? What makes the judge LLM trustworthy especially given the hallucination problem in LLMs. It would be useful to see where the peer and AC explanations differ, and quantitative metrics to support the claim of explainability.
- Robustness: It is not clear how the framework promotes robustness? Can perturbations in prompts not affect agentic systems? Is the claim based on the fact that a perturbation for some LLM will not affect another, if so do we have a measure of what part of perturbation space agentic systems are more robust to?  Are certain combinations of LLMs more robust than the others?
2. Similarly to the previous point, the conclusion says the framework is robust, but I see no experiments to validate this claim. This could be supported to some extent by showing the standard deviations in final ratings of each peer/AC, and the complete framework. However, in order to call the framework robust, there have to be evaluations of this system on common adversarial attacks to LLMs.
3. The work says on line 99-100 that the area chair should be a larger or more capable LM, but it is not clear why? Is using smaller peers justified because of efficiency? Given the ablation where a weaker area chair(section 5.2) also works, why use a larger one?
4.  I assume temperature was set to 1 to ensure variation in the responses when n>1, but is there a reason this performs better than setting temp=0 and using n=1?
5. Could there be analysis on the rating distributions of the framework when $n>1$?
6. I am unsure how $C_{final}$ is produced when n>1, because $S_i$ can be averaged but not sure how aggregations work for $C_i$.
7. If possible, it would be useful to see ablations on $n$, this could be done by choosing subsets from results of n=20 runs if the results are stored.
- The framework shows improvement in reasoning of multi-agent systems under the framework proposed.
- The evaluations/experiments presented are sound and well ablated.
- The error analysis section is useful and provides insights on where the framework could be improved further.
The paper proposes a hierarchical multi-LLM framework using a peer review structure for text evaluation and reasoning. The framework employs smaller models as peer reviewers and a larger model as an area chair to synthesize evaluations. The work demonstrates empirical improvements over existing approaches while providing rationales for other choices in the system including models and prompts.
- The paper highlights on lines 36-47 that there are not many works on using multiple-LLMs for evaluation, but there do exist some methods using multiple LLMs, for example: https://openreview.net/forum?id=FQepisCUWu and https://arxiv.org/html/2401.16788v1. The authors highlight very early and strongly the novelty of the work of using multiple-LLMs for evaluation but existence of the above works do bring in questions. The paper would benefit much more from better literature review, and inclusion of these other methods in the results section for a fair and faithful comparison.
- The work claims of developing a novel prompting schema by introducing "guidelines"(that can be auto-generated) and lists it as a main contribution. However, I think creating detailed prompts for specialized tasks is a well known prompt engineering practice, thus I do not see it as a substantial contribution.
- Several key claims require stronger support, especially on explainability and robustness, see questions below.
Yes
No ethics review needed.
Please see the weaknesses.
1. The paper is well written and the method is easy to reproduce;
2. Extending the evaluation tasks to reasoning tasks is a good generalization;
3. Some key issues, such as how to select models, how many models to use, are addressed though detailed ablation study.
This paper proposes a multi-agent framework, namely ReFeR, composed of a two-level hierarchical structure, where the two levels simulate the roles of reviewers and area chairs in the peer review process. The method has been compared comprehensively with mainstream approaches in terms of NLG evaluation tasks, multimodal evaluation tasks and for reasoning tasks.
1. The distinctions between this method and similar methods, such as multi-agent debate or multi-agent peer review, are not very clear. For example, debating or summarizing are merely different forms of prompts. In fact, for the multi-agent peer review method, if a reviewer can receive information from all other reviewers before refining their own review, they would essentially be playing a role very similar to that of an area chair.
2. The two-level hierarchical structure that summarizes through an area chair is not fundamentally different from ensemble methods like majority voting; it is merely implemented via prompts. Additionally, the difference between Turbo and Lite versions is simply one of 1 versus 20 rounds of integration.
3. The improvement in NLG evaluation is not particularly significant, and on reasoning tasks, the improvement is only more noticeable on AQuA, while it is not significant on other datasets. Moreover, the differences in performance may also be due to different model selections, such as the models used as peers not being entirely consistent with those used in other methods, leading to unfair comparisons.
4. The two-level hierarchical structure has not been theoretically proven to be necessarily better than debate methods; it seems more that different methods have different strengths, weaknesses, and application scenarios.
Yes
No ethics review needed.
Isn't the concept of using multiple LLMs to achieve better results in this context just derived from Mixture of Agents? Seems like an extension of the work; yet no reference.
How is the coverage of the instruction tuning dataset, does it scale?
Is ReFeR's main purpose to be a  framework to create instruction tuning sets? Or an evaluation system?
The originality is in adapting academic peer review processes to AI evaluation, offering a different approach to assessing LLM outputs. Quality is evident through comprehensive testing across reasoning tasks and multimodal outputs, with the framework showing consistent performance across different types of evaluation challenges. The paper is nicely structured, presenting both theoretical foundations and practical implementations through its two variants (ReFeR-Turbo and ReFeR-Lite). The framework's ability to handle both text and multimodal outputs while providing reasoned evaluations makes it particularly valuable for many real-world applications.
The paper introduces ReFeR, a hierarchical framework for evaluating AI-generated outputs that lack predefined correct answers, inspired by academic peer review processes. The framework consists of two main components: a Peer Review Body where multiple language models act as peer reviewers providing independent evaluations, and an Area Chair module where a more capable language model synthesizes these reviews into a final assessment.

The authors present two variants of the framework: ReFeR-Turbo, which generates multiple responses for higher accuracy but at greater computational cost, and ReFeR-Lite, which uses single responses for better efficiency. The framework also introduces enhanced evaluation guidelines and an auto-prompt generation system to improve the quality of assessments, making it a comprehensive solution for evaluating open-ended AI outputs.
The paper's main weakness is its unclear primary contribution, as it's difficult to distinguish how the ReFeR framework meaningfully differs from other multi-LLM evaluation systems beyond borrowing concepts from academic peer review (MoA). Additionally, the paper's organization is scattered across too many topics, with important elements like the instruction tuning dataset (shown in Figure 1) being buried in the appendix rather than properly discussed in the main text.
Yes
No ethics review needed.
Maybe I missed something, but how are the costs in Table 2-4 calculated?
- This work proposes an interesting framework for generative task evaluation inspired by the hierarchical peer review process with peer review agents from different LLMs and area chair evaluation. This provides a more systematic evaluation framework. It also gives more explainability for evaluation.
- ReFeR is flexible - it demonstrates effectiveness on various generative tasks in different modalities including text, image, and multimodal tasks. 
- There are analysis and ablation studies of the framework, which provides more insights on how to select models, how many models to use, etc.
- The paper is generally well written and easy to understand.
The paper introduces ReFeR, a hierarchical framework for evaluating generative models using a multi-agent system inspired by peer review. It leverages large language and vision-language models to assess outputs, providing more accurate, explainable evaluations than existing benchmarks. ReFeR is also good at reasoning tasks and offers two variants: ReFeR-Turbo (higher performance) and ReFeR-Lite (cost-effective). The framework demonstrates strong results across text, image, and reasoning evaluations, surpassing prior methods in both accuracy and efficiency.
- Even with the ReFeR-Lite variant, the framework still demands quite significant computational resources, making it still expensive as an evaluation tool.
- Another concern is that this framework can be considered as a hierarchical version of LLM-as-a-Judge with more carefully designed prompting, which may limit the novelty of this work.
Yes
No ethics review needed.
I would love to see the plots asked for in the Weaknesses box, and I will increase my score if appropriate. Additionally:

- I had some questions about how the G-Eval and Analyze-Rate baselines were configured; it would be good to move this information into the main text.
    - Is the backbone model matched to the AC model used in ReFeR? (GPT-4o-mini for NLG/Reasoning, GPT-4o for multimodal)?
    - Was n=20 for these?

- Similarly, was the single-agent CoT model matched to the AC model? Which agents were used in the multi-agent baseline?

- Figure 2 was a bit challenging to parse: which model is the AC model for the "Llama + Gemma + Nemo" point, and which models represent the "3 Peers + Mixtral" point?
- Automatic evaluation is an impactful problem, and I think the design space remains wide open. The grounding of evaluation in human annotations is great.

- The inclusion of an analysis section is great; I have some questions about Sections 5.1 and 5.3 (see Weaknesses / Questions), but Section 5.2 was very clear.

- I appreciate that the appendices are filled out, and that the authors included their code at submission time.
The authors propose an LLM-based automatic evaluation framework that draws inspiration from the academic peer review process: independent LLMs score and comment a generation, and then these scores, comments, and the original generation are fed to an "area chair" LLM that produces a final score. They evaluate this method by assessing correlation with human annotations on existing labeled datasets. They compare to existing LLM-automatic evaluation methods.
My main concern with this method is that in evaluations, the methods are not test-time compute matched. One explanation for ReFeR's stronger performance in Table 2 is that the system simply spends more inference compute than the other methods; this could mean that the peer review structure is not actually integral to performance. Since the paper's contribution is in the peer review structure, I think this is an important question to clarify.

- I appreciate that the authors compute a Cost column in Table 2, but this is only the AC cost. The authors justify this column as the monetary cost of querying an OpenAI API. However, in practice, this is not the only cost we have in evaluation: test-time compute (number of FLOPs processed) would be a more valid metric. (After all, in reality the authors are paying Together AI for inference of the open weight models as well.) I suspect that if we looked at test-time compute, ReFeR becomes significantly more heavyweight than other methods: after all, we need to inference 4 large language models!

- To make the performance - inference compute tradeoff more apparent, could the authors make a scatterplot with the corrected inference compute metric as the x-axis? An even stronger visualization would modulate n (or some other test-time compute parameter) for each method and plot how evaluation performance changes with increased test-time compute; we can then compare ReFeR to G-Eval, for example, by checking whether the ReFeR curve consistently sits above the G-Eval one.

    - I'd suggest drawing this up in a setting where the AC model is also an open-weight model; this way, we know all parameter counts.

I also would like to ask the authors to provide results for a few additional baselines: (1) just the AC, and (2) a simple average of the peer agent scores. Could the authors add GPT-4o-mini / GPT-4o (the AC model) and a simple average of scores as baseline rows in the "Peer Agents" section of Tables 2 and 3? I appreciate that Section 5.3 offers some insight into these questions, but I would like to directly see the human correlation scores of these baselines.
A framework to use Multiple AI agents for better evaluation of generative text or images and improved reasoning
@inproceedings{
anonymous2024refer,
title={ReFeR: Improving Evaluation and Reasoning through Hierarchy of Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=GDd5H92egZ},
note={under review}
}
Assessing the quality of outputs generated by generative models, such as large language models and vision language models, presents notable challenges. Traditional methods for evaluation typically rely on either human assessments, which are resource-intensive, or automatic metrics that often show a low correlation with human judgment. Another common approach is to use deep learning systems, which not only consume a substantial amount of compute and time but also require extensive training data. In this study, we introduce a tuning-free framework called ReFeR, designed to evaluate generative outputs, including both text and images, by leveraging a 2-level hierarchy of LLMs and VLMs themselves.  We rigorously evaluate our framework, ReFeR, across four diverse evaluation tasks. The framework not only improves the accuracy of these evaluations, surpassing previous benchmarks but also generates constructive feedback. Interestingly, the framework is also applicable to reasoning tasks. Experiments on four reasoning tasks demonstrate superior collective reasoning abilities of the framework. We present two variants of the framework: ReFeR-Turbo, optimized for accelerated performance, and ReFeR-Lite, offering a more cost-effective solution. ReFeR-Lite is $\sim7.7\times$ more efficient while being comparably accurate to ReFeR-Turbo.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Multi-agent framework reasoning evaluation multimodal hierarchy LLMs VLMs
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/915b2cb2f7c1889206c79fc20d605ee24c04975e.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
ReFeR: Improving Evaluation and Reasoning through Hierarchy of Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission