updated agent, critic, aggregator prompts

app.py

@@ -168,15 +168,14 @@ def create_feedback(review, pdf_text, agent_prompt, model):
 
     return feedback
 
-def pick_best_review(feedback_list, agent_prompt, review, pdf_text, model):
-    prompt = f"You are given a list containing output from multiple iterations of the same agent. The agent was given the following prompt: {agent_prompt}. OK the agent prompt is done. Now, your task is to go through each iteration of feedback and pick the best overall feedback for the review. This can be portions of feedback from each iteration, or one set of feedback in its entirety. Please return the feedback in the exact format and wording it appears in. You will be given the original review, the list of feedback from the different agents separated by the '^' character, and the paper the review is about."
+def aggregator(feedback_list, aggregator_prompt, agent_prompt, review, pdf_text, model):
 
     messages = [{
         "role": "user",
         "content": [
       {
         "type": "text",
-        "text": prompt
+        "text": aggregator_prompt
       },
       {
         "type": "text",
@@ -291,165 +290,113 @@ def critic(review, feedback, pdf_text, critic_prompt, model):
     return revised_feedback
 
 agent_prompt = """
-You are given a peer review of a machine learning paper submitted to a top-tier ML conference on OpenReview.
-First, you will read the text of the review and then the paper about which it was written.
-Give feedback to the reviewer so that it becomes a high-quality review.
-Here is a checklist. For each item, determine if the review passes or fails. If it fails, provide feedback accordingly. Follow the below guidelines for your feedback:
-
-1. The reviewer does not ask for anything that is OBVIOUSLY already present in the paper. You should be ABSOLUTELY certain the reviewer made an error, otherwise do NOT comment on their point.
-If you find such a case, quote the specific statement from the review and provide the exact quote from the paper that directly addresses or contradicts it. 
-Ensure that the quote from the paper specifically answers the claim or request made by the reviewer, and explain why the selected quote is relevant.
-If the reviewer has a question, you do not need to respond. If you respond, point to a specific section or quote from the paper and ask “does this provide what you were looking for?”
-2. The review does not make any vague, unjustifiable, or unsupported claims. 
-If there are any vague comments in the review, ask the reviewer to be more SPECIFIC and DETAILED.
-3. If the reviewer claims the paper isn't novel, they should explain why and specify the paper(s) they believe this work is similar to. 
-If they have not already done this, ask the reviewer to do so.
-4. The reviewer does not make any personal attacks against the paper and/or author(s). 
-For example, they call the work "incompetent" without justifying why. 
-If they do, suggest the reviewer revise their feedback to remove those attacks.
-
-Be concise, limiting your response to 2-3 sentences per checklist item. Do not repeat anything that the reviewer already included in their review, and do not summarize your feedback at the end.
-
-Here are some examples of good responses to reviews that may fall under each of these categories.
-
-1. A question or claim that is either already answered in the paper or contradicts what is provided in the paper.
-
-For such cases, we would like to point the reviewer specifically to the relevant part of the paper through direct quoting. Only send a quote if it verbatim exists in the paper or review.
-
-Example 1:
-
-**Reviewer Comment:** *..In Figure 4, the efficiency experiments have no results for Transformer models, which is a key limitation of the paper.I*
-
-**Feedback to reviewer:** You may want to check Section 3, Figure 5 of the paper which has the Transformer results. See: “In Transformers, the proposed technique provides 25% relative improvement in wall-clock time (Figure 5)”.
-
-Example 2: 
-**Reviewer Comment:** *Figure 2. Are people forced to select a choice or could they select 'I don't know'? Did you monitor response times to see if the manipulated images required longer times for individuals to pass decisions? In Appendix A, you mention “the response times will also be released upon publication”, however, I do not see further information about this in the paper.*
-
-As the reviewer already refers to the key part of the paper with quotes and asks a question that is not answered in the paper, we do not need to give feedback to this comment.
-
-2. Lack of clarity or justification for criticism: The reviewer criticizes the paper without providing a concrete justification. This results in points that are not actionable or harder to respond to. 
-
-For such cases, we would like to nudge the reviewer to justify their claim.
-
-Example 1:
-**Reviewer Comment:** *A key limitation is that this paper motivates from the TTT perspective, but no TTT experiments are performed and compared, e.g., comparing with (Sun et al., 2020).*
-
-**Feedback to reviewer:** In general, a “TTT experiment” is not a broadly known terminology. It would be very helpful if you could describe what specific experiment you are referring to, and why it is important for this paper.
-
-Example 2: 
-**Reviewer Comment:** *W1: Many of the key pieces of this work have been considered before separately. I think some additional discussion of related work is needed.
-
-W2: It appears that the linear mode connectivity results may be somewhat brittle.*
-
-**Feedback to reviewer:** For W1, It would be very helpful to specify or cite what key pieces are considered in earlier works, how they relate to the current work, and specify what is missing. For W2, could you elaborate on why you see the results as brittle and what would be convincing evidence so that the feedback is more actionable?
-
-Example 3: 
-**Reviewer Comment:** *The data and model size are not large-scale, thus the paper will not be impactful.*
-
-**Feedback to reviewer:** Please consider putting this claim in context. If possible, please discuss why you consider the experiments to be not large-scale using precedents, why it is important to have larger-scale experiments in this context, and how it would inform or improve the manuscript.
-
-3. Novelty claims: The reviewer claims a lack of novelty without providing concrete justification.
-
-For this case, we would like to nudge the reviewer to justify the claim, by prompting them to provide the most relevant references, the relationships, and specifying similarities or differences.
-
-Example 1:
-**Reviewer Comment:** *.. The paper’s novelty is limited considering the ICLR standards, as there are very close works [1,2,3]. *
-
-**Feedback to reviewer:** Please consider the reasons for why the novelty is limited, and specify what ICLR standards are in this context. In particular, it would be very helpful to all parties involved if you give examples of the closest papers, their similarities, and differences with the methods or results in the current paper.
-
-Example 2: 
-**Reviewer Comment:** *.. DASHA is a mash-up between MARINA and existing distributed nonconvex optimization methods. Other than the fact that three variants of DASHA get rid of the uncompressed synchronization in MARINA, this reviewer could not pinpoint a difference between MARINA and DASHA. As such, the main novelty of this work seems to be in terms of theoretical analysis of MARINA when the uncompressed synchronization step is removed. The authors could have done a better job of clarifying where does this novelty lie in the analysis (e.g., pinpointing the key analytical approaches in the lemma that helped improve the analysis)*
-
-As the novelty claim is already well-justified and an actionable question is asked, we do not need to give feedback to this review.
-
-4. Personal attack, ad hominem: The review contains an attack on the authors. This can be about the personality, the knowledge, or the experience of the authors. 
-
-For this case, we would like to kindly warn the reviewer about their comment. 
-
-Example 1: 
-**Reviewer Comment:** *.. The authors clearly do not live in the real world and do not care about people or downstream effects of their research. *
-
-**Feedback to the reviewer:** We kindly suggest you revise this comment, as it includes remarks about the personalities or intents of the authors.
+You are given a peer review of a machine learning paper submitted to a top-tier ML conference on OpenReview. Your task is to provide constructive feedback to the reviewer so that it becomes a high-quality review. You will do this by evaluating the review against a checklist and providing specific feedback about where the review fails. 
+
+Here are step-by-step instructions:
+1. Read the review and the paper 
+    - Carefully read through the text of the review,
+    - Then, read the paper about which the review was written to understand its content and context.
+
+2. Evaluate every comment in the review:
+   - Focus on comments made in the "weaknesses" or "questions" sections of the review. Ignore the "summary" and "strengths" sections.
+   - For each comment, evaluate it against the following checklist. Examples are provided of good responses.
+
+     Checklist:
+     1. Check if the reviewer requests something that is OBVIOUSLY already present in the paper. Only proceed if you are ABSOLUTELY certain the reviewer made an error, otherwise do NOT comment on their point. If you find such a case, point the reviewer specifically to the relevant part of the paper through direct quoting. Only send a quote if it verbatim exists in the paper or review. Ensure that the quote from the paper specifically answers the claim or request made by the reviewer, and explain why the selected quote is relevant.
+        - Example 1: 
+          - **Reviewer Comment:** *..In Figure 4, the efficiency experiments have no results for Transformer models, which is a key limitation of the paper.I*
+          - **Feedback to reviewer:** You may want to check Section 3, Figure 5 of the paper which has the Transformer results. See: “In Transformers, the proposed technique provides 25% relative improvement in wall-clock time (Figure 5)”.
+        - Example 2:
+            - **Reviewer Comment:** *Figure 2. Are people forced to select a choice or could they select 'I don't know'? Did you monitor response times to see if the manipulated images required longer times for individuals to pass decisions? In Appendix A, you mention “the response times will also be released upon publication”, however, I do not see further information about this in the paper.*
+            - As the reviewer already refers to the key part of the paper with quotes and asks a question that is not answered in the paper, we do not need to give feedback to this comment.
+
+     2. Look for any vague or unjustifiable claims in the review. This results in points that are not actionable or harder to respond to. For such cases, we would like to nudge the reviewer to provide more specific details and justify their claim.
+        - Example 1: 
+          - **Reviewer Comment:** *A key limitation is that this paper motivates from the TTT perspective, but no TTT experiments are performed and compared, e.g., comparing with (Sun et al., 2020).*
+          - **Feedback to reviewer:** In general, a “TTT experiment” is not a broadly known terminology. It would be very helpful if you could describe what specific experiment you are referring to, and why it is important for this paper.
+        - Example 2:
+            - **Reviewer Comment:** *W1: Many of the key pieces of this work have been considered before separately. I think some additional discussion of related work is needed. W2: It appears that the linear mode connectivity results may be somewhat brittle.*
+            - **Feedback to reviewer:** For W1, It would be very helpful to specify or cite what key pieces are considered in earlier works, how they relate to the current work, and specify what is missing. For W2, could you elaborate on why you see the results as brittle and what would be convincing evidence so that the feedback is more actionable?
+        - Example 3:
+            - **Reviewer Comment:** *The data and model size are not large-scale, thus the paper will not be impactful.*
+            - **Feedback to reviewer:** Please consider putting this claim in context. If possible, please discuss why you consider the experiments to be not large-scale using precedents, why it is important to have larger-scale experiments in this context, and how it would inform or improve the manuscript.
+     3. If the reviewer claims the paper lacks novelty, ensure they specify why, including references to similar work. If they haven't, we would like to nudge the reviewer to justify the claim, by prompting them to provide the most relevant references, the relationships, and specifying similarities or differences.
+        - Example 1: 
+            - **Reviewer Comment:** *.. The paper's novelty is limited considering the ICLR standards, as there are very close works [1,2,3]. *
+            - **Feedback to reviewer:** Please consider the reasons for why the novelty is limited, and specify what ICLR standards are in this context. In particular, it would be very helpful to all parties involved if you give examples of the closest papers, their similarities, and differences with the methods or results in the current paper.
+        - Example 2:
+            - **Reviewer Comment:** *.. DASHA is a mash-up between MARINA and existing distributed nonconvex optimization methods. Other than the fact that three variants of DASHA get rid of the uncompressed synchronization in MARINA, this reviewer could not pinpoint a difference between MARINA and DASHA. As such, the main novelty of this work seems to be in terms of theoretical analysis of MARINA when the uncompressed synchronization step is removed. The authors could have done a better job of clarifying where does this novelty lie in the analysis (e.g., pinpointing the key analytical approaches in the lemma that helped improve the analysis)*
+            - As the novelty claim is already well-justified and an actionable question is asked, we do not need to give feedback to this review.
+     4. Identify any personal attacks or inappropriate remarks made by the reviewer. This can be about the personality, the knowledge, or the experience of the authors. For example, they call the work "incompetent" without justifying why. For this case, we would like to kindly warn the reviewer about their comment and politely suggest they revise their language.
+        - Example 1: 
+            - **Reviewer Comment:** *.. The authors clearly do not live in the real world and do not care about people or downstream effects of their research. *
+            - **Feedback to the reviewer:** We kindly suggest you revise this comment, as it includes remarks about the personalities or intents of the authors.
+
+3. Provide feedback:
+    - For each comment that fails according to the checklist, write concise feedback in the following format:
+        - Comment: {{the comment of interest}}
+        - Feedback: {{your short feedback}}
+    - If you do not identify any issues with a comment, do not include it in your feedback list.
+    - If you find no issues in the review at all, respond with: "Thanks for your hard work!"
+
+Remember:
+- Be concise, limiting your feedback for each comment to 1-2 sentences.
+- Do not summarize your feedback at the end or include a preamble at the beginning.
+- Do not repeat anything the reviewer already included in their review.
+- Do not mention that you are using a checklist or guidelines.
+- DO not address the authors at all or provide suggestions to the authors. You are only giving feedback to the reviewer.
+"""
 
-Your instructions: Go through all comments in the review. Check each comment using the above guidelines. Only address issues with the review that fail according to the above guidelines. If the review is in a format where it has a "summary" and "strengths" section, DO NOT address those sections. Only address weaknesses or questions.  Make sure you only address issues with the review.
-Your feedback will be sent directly to reviewers. Follow the below format when sending your review:
+critic_prompt = f"""
+You are given a list of feedback about a peer review of a machine learning paper submitted to a top-tier ML conference on OpenReview. The aim of the feedback is to guide a reviewer to make the review high-quality. Your task is to edit the feedback for correctness and clarity.
+
+Here are step-by-step instructions:
+1. Read the review, the feedback list, and the paper 
+    - Carefully read through the text of the review
+    - Then, read the feedback list provided for that review
+    - Finally, read the paper about which the review was written to understand its content and context.
+
+2. Evaluate every piece of feedback in the feedback list:
+   - For each feedback item, it is imperative that you evaluate the correctness of the feedback. If there is a quote in the feedback, ensure that the quote appears VERBATIM in the paper. You need to check every quote and factual claim in the feedback and edit for correctness, as it is imperative all the feedback is correct.
+   - For each feedback item, evaluate if it is clear. You should make sure it would not confuse or frustrate the reviewer who reads it.
+   - Make sure every comment-feedback pair addresses an issue with the review. If it does not, remove it.
+
+3. Edit comments based on evaluations:
+    - Do NOT ADD any new points unless the previous feedback OBVIOUSLY missed something important. 
+    - For each comment that needs editing, re-write the feedback concisely in the following format:
+        - Comment: {{the comment of interest}}
+        - Feedback: {{your short feedback}}
+    - If you do not identify any issues with a comment-feedback pair, do not modify it.
+    - If the review has no issues at all, simply respond with: "Thanks for your hard work!"
+
+Remember:
+- Be concise, limiting your feedback for each comment to 1-2 sentences.
+- Do not summarize your feedback at the end or include a preamble at the beginning.
+- Do not repeat anything the reviewer already included in their review.
+- Do not mention that you are using a checklist or guidelines.
+- DO not address the authors at all or provide suggestions to the authors. You are only giving feedback to the reviewer.
+
+Here are the guidelines that were followed to generate the feedback list originally; you should adhere to these guidelines: {agent_prompt}.
+"""
 
-- Comment: {{the comment of interest}}
-- Feedback: {{write your short feedback}}
+aggregator_prompt = f"""
+You are given multiple lists of feedback about a peer review of a machine learning paper submitted to a top-tier ML conference on OpenReview. The aim of the feedback is to guide a reviewer to make the review high-quality. Your task is to aggregate the lists of feedback into one list.
 
-and send a list of comments and feedbacks. If you do not identify issues for a comment, do not add it to the list or send feedback. DO NOT mention that there is a checklist or guidelines.
+Here are step-by-step instructions:
+1. Read the review, the multiple feedback lists, and the paper 
+    - Carefully read through the text of the review
+    - Then, read each of the feedback lists provided for that review
+    - Finally, read the paper about which the review was written to understand its content and context.
 
-If you cannot identify any issues, respond with "Thanks for your hard work!"
-"""
+2. For all feedback lists, aggregate them into one list with the best comment-feedback pairs from each list.
+    - For each comment-feedback pair in the multiple lists that are similar, determine which provides the best feedback and keep only that pair.
+    - If there are unique comment-feedback pairs in the multiple lists, criticially determine if it is an essential piece of feedback needed to improve the review. If it us unncessary or redundant, remove the comment-feedback pair.
+    - You should end up with one feedback list that has no repeated comments from the review and that is high quality.
+    - Return the feedback list in the format you received it in, where the pairs are formatted as:
+        - Comment: {{the comment of interest}}
+        - Feedback: {{your short feedback}}
 
-critic_prompt = """
-You are given feedback about a peer review of a machine learning paper submitted to a top-tier ML conference on OpenReview. The aim of the feedback is to make the review high-quality.
-First, you will read the original review, then you will read the feedback, and then you will read the paper the review was written about.
-You need to edit the feedback for correctness, removing anything that is incorrect and revising anything that needs to be fixed.
-You should also edit the feedback for clarity, removing anything that may frustrate or confuse a reviewer.
-
-The original feedback was written according to the following checklist. For each item, the agent determined if the review passed or failed. If it failed, the agent provided feedback accordingly. Here is the checklist:
-
-1. The reviewer does not ask for anything that is OBVIOUSLY already present in the paper. You should be ABSOLUTELY certain the reviewer made an error, otherwise do NOT comment on their point.
-If you find such a case, quote the specific statement from the review and provide the exact quote from the paper that directly addresses or contradicts it. 
-Ensure that the quote from the paper specifically answers the claim or request made by the reviewer, and explain why the selected quote is relevant.
-If the reviewer has a question, you do not need to respond. If you respond, point to a specific section or quote from the paper and ask “does this provide what you were looking for?”
-2. The review does not make any vague, unjustifiable, or unsupported claims. 
-If there are any vague comments in the review, ask the reviewer to be more specific.
-3. If the reviewer claims the paper isn't novel, they should explain why and specify the paper(s) they believe this work is similar to. 
-If they have not already done this, ask the reviewer to do so.
-4. The reviewer does not make any personal attacks against the paper and/or author(s). 
-For example, they call the work "incompetent" without justifying why. 
-If they do, suggest the reviewer revise their feedback to remove those attacks.
-
-The agent was told to “be concise, limiting your response to 2-3 sentences per checklist item. Do not repeat anything that the reviewer already included in their review, and do not summarize your feedback at the end.”
-
-Here are some examples of good responses to reviews that the agent was given that may fall under each of these categories.
-
-1. A question or claim that is either already answered in the paper or contradicts what is provided in the paper.
-For such cases, we would like to point the reviewer specifically to the relevant part of the paper through direct quoting. Only send a quote if it verbatim exists in the paper or review.
-Example 1:
-**Reviewer Comment:** *..In Figure 4, the efficiency experiments have no results for Transformer models, which is a key limitation of the paper.I*
-**Feedback to reviewer:** You may want to check Section 3, Figure 5 of the paper which has the Transformer results. See: “In Transformers, the proposed technique provides 25% relative improvement in wall-clock time (Figure 5)”.
-Example 2: 
-**Reviewer Comment:** *Figure 2. Are people forced to select a choice or could they select 'I don't know'? Did you monitor response times to see if the manipulated images required longer times for individuals to pass decisions? In Appendix A, you mention “the response times will also be released upon publication”, however, I do not see further information about this in the paper.*
-As the reviewer already refers to the key part of the paper with quotes and asks a question that is not answered in the paper, we do not need to give feedback to this comment.
-
-2. Lack of clarity or justification for criticism: The reviewer criticizes the paper without providing a concrete justification. This results in points that are not actionable or harder to respond to. 
-For such cases, we would like to nudge the reviewer to justify their claim.
-Example 1:
-**Reviewer Comment:** *A key limitation is that this paper motivates from the TTT perspective, but no TTT experiments are performed and compared, e.g., comparing with (Sun et al., 2020).*
-**Feedback to reviewer:** In general, a “TTT experiment” is not a broadly known terminology. It would be very helpful if you could describe what specific experiment you are referring to, and why it is important for this paper.
-Example 2: 
-**Reviewer Comment:** *W1: Many of the key pieces of this work have been considered before separately. I think some additional discussion of related work is needed.
-W2: It appears that the linear mode connectivity results may be somewhat brittle.*
-**Feedback to reviewer:** For W1, It would be very helpful to specify or cite what key pieces are considered in earlier works, how they relate to the current work, and specify what is missing. For W2, could you elaborate on why you see the results as brittle and what would be convincing evidence so that the feedback is more actionable?
-Example 3: 
-**Reviewer Comment:** *The data and model size are not large-scale, thus the paper will not be impactful.*
-**Feedback to reviewer:** Please consider putting this claim in context. If possible, please discuss why you consider the experiments to be not large-scale using precedents, why it is important to have larger-scale experiments in this context, and how it would inform or improve the manuscript.
-
-3. Novelty claims: The reviewer claims a lack of novelty without providing concrete justification.
-For this case, we would like to nudge the reviewer to justify the claim, by prompting them to provide the most relevant references, the relationships, and specifying similarities or differences.
-Example 1:
-**Reviewer Comment:** *.. The paper’s novelty is limited considering the ICLR standards, as there are very close works [1,2,3]. *
-**Feedback to reviewer:** Please consider the reasons for why the novelty is limited, and specify what ICLR standards are in this context. In particular, it would be very helpful to all parties involved if you give examples of the closest papers, their similarities, and differences with the methods or results in the current paper.
-Example 2: 
-**Reviewer Comment:** *.. DASHA is a mash-up between MARINA and existing distributed nonconvex optimization methods. Other than the fact that three variants of DASHA get rid of the uncompressed synchronization in MARINA, this reviewer could not pinpoint a difference between MARINA and DASHA. As such, the main novelty of this work seems to be in terms of theoretical analysis of MARINA when the uncompressed synchronization step is removed. The authors could have done a better job of clarifying where does this novelty lie in the analysis (e.g., pinpointing the key analytical approaches in the lemma that helped improve the analysis)*
-As the novelty claim is already well-justified and an actionable question is asked, we do not need to give feedback to this review.
-
-4. Personal attack, ad hominem: The review contains an attack on the authors. This can be about the personality, the knowledge, or the experience of the authors. 
-For this case, we would like to kindly warn the reviewer about their comment. 
-Example 1: 
-**Reviewer Comment:** *.. The authors clearly do not live in the real world and do not care about people or downstream effects of their research. *
-**Feedback to the reviewer:** We kindly suggest you revise this comment, as it includes remarks about the personalities or intents of the authors.
-
-
-Your instructions: Do not change the format of the feedback, only edit for content clarity and correctness. DO NOT ADD any new points unless the previous feedback OBVIOUSLY missed something important. You need to check every quote and factual claim in the feedback and edit for correctness, as it is imperative all the feedback is correct. If the review is in a format where it has a "summary" and "strengths" section, DO NOT address those sections. Make sure you only address issues with the review.
-Your feedback will be sent directly to reviewers. DO NOT address the authors at all or provide suggestions to the authors. You are only giving feedback to the reviewer. Follow the below format when sending your review:
-- Comment: {{the comment of interest}}
-- Feedback: {{write your short feedback}}
-and send a list of comments and feedbacks. If you do not identify issues for a comment, do not add it to the list or send feedback. DO NOT mention that there is a checklist or guidelines, and do NOT mention that this is the edited feedback. Do not add a preamble, go right to the comments and feedback.
-If you have no feedback, then respond with "Thanks for your hard work!"
+Here are the guidelines that were followed to generate the feedback list originally: {agent_prompt}.
 """
 
 # Set page to wide mode
@@ -506,7 +453,7 @@ if user_input:
         for _ in range(iterations):
             feedback = create_feedback(review, pdf_text, agent_prompt, model)
             feedback_list.append(feedback)
-        best_feedback = pick_best_review(feedback_list, agent_prompt, review, pdf_text, model)
+        best_feedback = aggregator(feedback_list, aggregator_prompt, agent_prompt, review, pdf_text, model)
     else:
         best_feedback = create_feedback(review, pdf_text, agent_prompt, model)