Preference Data Annotation with Guided Density Ratios
RLHF;preference training;reward model;human preference;preference alignment
Yes
No ethics review needed.
Please see Weaknesses section. 

I believe that addition of answers to the above questions can help with a stronger submission.
The paper does a good job in explaining their contribution, and the examples are helpful. The direction of LLM as a preference / human proxy is topical and being investigated by various groups - which makes it aligned to the venue.
The paper interprets the use of a strong v/s weak LLM as a human preference proxy. However, their main contribution is designing prompt templates with relevant domain specific exemplars (IC examples) and domain specific system prompts - to show that the resulting density ratio is more helpful.
I find the work to be an incremental advancement over previous discussions on using LLMs as a reward proxy. The key contribution of the work is the manually designed prompt templates, T(x), which are composed of system and in-context examples - both of which are domain dependent. The work's key argument is to favor a domain specific prompt design that influences density ratio - with limited discussions on why this template must work. 

Few questions to the authors : 
1. What is the cost of designing these templates? How can one extend this work to an arbitrary domain where the preferences may not be "safety". 
2. In line 048, authors say that human preferences can be complicated. How are the preferences anymore complicated or undefined when they they pen down the preferences as part of their system prompt. Can the authors point out to examples where these prompt templates helped with other user preferences not explicitly mentioned in the prompt. 
3. Can the authors compare the cost of prompt design with collecting more feedback? 
4. What are the guarantees that this prompt template won't work adversarially to the model. For instance, its possible that the system prompt conflicts with some preferences specified in the query. What are some alternatives to the system prompt that the authors considered? The authors provide some discussion in A.1, however, the following questions remain unanswered :  how did the authors generate SYSTEM1, 2... versions; what are the domain specific principles and how were those generated?; what other principles did the authors experiment with? Which principles, while "plausible" and "aligned" with typical human preferences did not yield expected results. 
5. Many a times IC examples have hampered model's performance. The authors discuss ICL example selection strategy in A.1, but the discussion seems to to justify the use of IC examples in their setup anecdotally. 
6. The argument for why a baseline calibration is needed (lines 205 - 211) seem anecdotal. Can the authors provide stronger theoretical / empirical result on what are the properties of this weaker model for such a calibration, and what must the relationship between weaker and stronger model be in the scope of their work.
Yes
No ethics review needed.
1. Could the density ratio approach be adapted to work with multi-modal data, like vision language models / LLaVA, or is it inherently limited to language models?
1. The results on RewardBench and other benchmarks demonstrate that the proposed method can compete with or even outperform larger, more resource-intensive models, making it a cost-effective solution for preference alignment.

2. The paper includes comprehensive experiments across multiple domains and benchmarks, providing robust evidence for the effectiveness of the method.

3. The paper introduces a scalable, training-free preference annotation method using density ratios, a concept that could significantly reduce the cost and time typically required for human feedback data collection.
The paper proposes a scalable, training-free method for preference data annotation in large language models by using guided density ratios between pre-trained models. This approach, which employs domain-specific prompts and few-shot examples, enhances alignment in areas like safety and reasoning without requiring costly human feedback. The method achieves competitive results on benchmarks such as RewardBench and MT-Bench, demonstrating its potential as an effective and resource-efficient alternative for preference tuning in LLMs.
1. While prompt-based guidance improves performance, designing effective prompts and in-context examples for different domains could be labor-intensive and may not generalize well across all tasks.

2. I think evaluation across different model size should be included. For example, including 7B, 13B, and 70B.
Yes
No
No ethics review needed.
- How to ensure the annotate quality in the prompt routing for category-specific prompts and preference tunning data creation?
+ The paper explores the challenge of data scarcity in preference tuning, a significant question in the LLM era.

+ The author proposes a novel data annotation technique based on the prompt density ratio, which has advanced community skills and tunning-free advantages.

+ Extensive experiments include multiple settings on the cutting-edged open-source and close-resource MLLM to improve the solidity of the paper.
The paper investigates the preference tunning in LLM and proposes the challenge that it depends on paired human feedback data and the collection is costly in time and resources. Then the authors propose a novel data annotation technique that improves the density ratio rewards annotation accuracy and enhances the effectiveness of preference tuning using a new annotated dataset.
- While the proposed prompt-based density ratio method exhibits some improvements, the design of prompt templates for system prompts and few-shot prompts is widely used in broad scenarios with less innovation.

- Prompt-density ratio compared to vanilla-density ratio decreases the model’s bias, but if the model itself with some bias, it will accumulate to the updated modal, it should expand some discussion about the difference of bias between source and target model.

- The experiment section should expand a paragraph to summarise the setting and the experiment description is unclear. While I was confused by some of the questions listed below. 

What is the generative reward in Table 1? 

Why the second to last only include safety and reasoning? 

Why are the results of NousResearch/Nous-Hermes-2-Mistral-7B-DPO not included in the Table, which is also an instruction-tuned model? 

Why is only one setting included with a base or without a base that corresponds to two different weak models?

Does the experiment include a domain-specific prompt that should also include the comparison?

- There is a lack of ablation studies on the functionality of system prompts and few-shot prompts.

- Some sentences contain long-tail expressions, and several equations need improvement, particularly by adding appropriate punctuation marks.
This paper finds that injecting domain-specific prompts improves the controllability and overall efficacy of density ratio reward. It boosts RewardBench by 4 points, with major gains in safety and reasoning scores.
@inproceedings{
anonymous2024preference,
title={Preference Data Annotation with Guided Density Ratios},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=nHenODN9je},
note={under review}
}
Preference tuning has become a standard step of modern LLM post-training. Usually, it requires paired human feedback data or preference classifiers trained on such data, where the data collection is costly in time and resources.
This paper proposes a data annotation technique that takes the prompt-guided density ratio between off-the-shelf LLMs to serve as proxy of human preference with no training needed. We show that by adding descriptions of preference and domain specific few-shot examples before the user query (e.g. a detailed definition of safety plus an example), we can significantly improve density ratio rewards' annotation accuracy. Our final method reaches a score of 82.6 on RewardBench, where prompt injection improves the Safety domain from 82 to 91 and the Reasoning domain from 74 to 90. We then perform preference tuning using data annotated by density-ratio reward from a 7B model, aligning a Llama 3 8B instruct model to achieve an 37% WinRate on ArenaHard, 41% Length Controlled win-rate on AlpacaEval 2.0, and 8.0 on MT-Bench.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
RLHF preference training reward model human preference preference alignment
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/e2b485e7574521034da2327fd9ffc91286df4eeb.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Preference Data Annotation with Guided Density Ratios
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission