PersonalLLM: Tailoring LLMs to Individual Preferences
Personalization;LLM;Alignment;benchmark;dataset;reinforcement learning from human feedback;language models;RLHF;preferences
Yes
Yes, Discrimination / bias / fairness concerns
see the weakness
1. PersonalLLM provides a way to enhance the personalization of LLMs, which is an impactful direction to enhance the user experience. 

2. The benchmark includes extensive open-ended prompts with responses from state-of-the-art LLMs. 

3. The paper highlights the use of meta-learning to address data sparsity issues by leveraging historical interactions, which is crucial for real-world applications where personalized models lack sufficient user-specific data.
The paper introduces PersonalLLM, a public benchmark designed to personalize Large Language Models (LLMs) to better align with individual user preferences. The benchmark focuses on simulating diverse personal preferences using a set of pre-trained reward models. The dataset consists of open-ended prompts paired with multiple high-quality LLM responses, and the goal is to optimize personalization by leveraging historical user data. Basic baselines, including in-context learning and meta-learning, are explored to showcase the utility of this benchmark, setting the stage for future research into personalization algorithms for LLMs.
1. The personal preference models used to simulate diverse user preferences are not convincing enough to represent real users. First, it is difficult to verify whether the linear combination of scores from reward models aligns with the distribution of user rewards in the real world. Second, the candidate responses generated by LLMs may not cover real-world user-specific responses, making it challenging for LLMs to learn user-specific preferences or align with user-specific backgrounds. For instance, users may have particular preferences or habits that general reward models inherently struggle to account for when providing accurate rewards.

2. The paper lacks an overarching figure that illustrates the construction logic of the dataset and what the samples within the dataset look like.

3. The comparison of the paper with other relevant personalized LLM benchmarks, such as the LaMP dataset.

4. Some related concepts are not clearly explained, such as 'interaction history', 'preference data', and 'user data,' which are not well defined.
Yes
Yes, Responsible research practice (e.g., human subjects, data release)
Answering and solving the weakness questions clearly can greatly help the reviewer target the focus of the paper. For the reviewer, these issues require a lot of time to carefully polish the paper before they can be completed.  In addition, the review would ask:
What is the relationship between PERSONALLLM and recommender system? Is it a replacement of existing ones or a more general preferenc-based system incuding RS? Why?
1. The paper uses multiple LLMs to generate various responses to improve the confidence of dataset.
2. The paper provides a specific analysis of the dataset.
The paper aims to propose a dataset called PERSONALLLM for the personalization AI area, which contains users’ preference illustrated by a prompt with eight responses. Specifically, the user responses are built up by various LLMs, e.g., GPT4, Claude 3.

The authors then propose in-context learning and meta-learning methods as baselines for two scenarios from PERSONAL. The results show that there is much room for improvement in solving the personalization problem in the proposed PERSONAL.
1. The paper is unclear about what the preference is in the data; is it user preference of items in recommender systems or any replacement of NLP tasks or others?
2. The paper is uclear about how the PERSONALLLM is formulated, the author presented the reward model, but how it is trained/built up.
3. The author illustrates the heter preference PERSONALLLM involves in which differs from the home ones, but how these two preferences demonstrate is not clear.
Yes
It's worth it to double-check that including the LLM responses in a dataset is within the relevant terms of use -- my impression is that generally they are, but it should be double-checked.
Yes, Legal compliance (e.g., GDPR, copyright, terms of use)
1) Are there other sources of high-quality reward functions that can be used?
2) Were the leading LLMs used to sample the 8 preferences prompted with personas?
Originality: The paper proposes (as far as I know) an original method for generating diverse user preferences.
Quality: The paper both creates a high-quality dataset, as well as empirically validates that its methodology creates diverse preferences at least as diverse as a persona-based method.
Clarity: The paper is clearly written.
Significance: The paper establishes a dataset and methodology for generating diverse user preferences, which is very important for studying LLM personalization.
This paper builds a dataset of open-ended prompts and high-quality responses where users might be expected to have different preferences, a method of sampling direct different user preferences based on reward models, and proposes different algorithms for personalization using data across multiple users. In addition, they empirically validate that their proposed method of sampling user preferences beats a baseline persona-based method for generating diverse user preferences.
1) The paper uses reward models from a leaderboard (as opposed to fine-tuning to persona data or something), which means that the reward models are all high-quality, but may result in reward models which are less distinct from each other than they might otherwise be. The paper clearly justifies this as not preventing their resampling method from reaching higher diversity than persona-based prompting, but are there other sources of high quality reward functions that might be more different from each other?
2) Similarly, were the leading LLMs used to sample the 8 preferences prompted with personas? The different LLMs might be somewhat more similar to each other than they need to be, but of course resampling the dataset could be quite expensive, and the dataset is quite valuable as is.
Yes
No ethics review needed.
* In line 76 it says “in doing so we are able to simulate an entire user base”. On the other hand it says in line 102 that “We do not claim our simulated personal preference models provide a high-fidelity depiction of human behavior”, so this may be a bit confusing and you may want to rephrase these statements. After reading the first one I was hoping for some evaluation of how realistic the simulated users are. This is actually done in “Comparison to Human Preferences” in Section 3, so I guess you are doing some of that? If the goal is to obtain high *coverage* rather than matching the distribution of users, perhaps this can be made explicit and possibly evaluated against real user behavior? Perhaps some measure of support instead of Wasserstein? It would be also interesting to compare the results in Figure 5 to those from standard personas baselines.
Actually, if the goal is coverage then random preferences should give better coverage, but are probably not very useful, so just optimizing coverage doesn’t seem to be a good objective.
Can you please clarify the objective here?
* Another potentially interesting baseline is to have each user choose one of the rewards, a hard choice instead of a weighted sum. There will only be 10 user “types”, so it may be interesting to see how the results change in that case.
* Sometimes there are long multi-line sentences that could be simplified to improve readability and flow. It is easier to read a paper that has no sentences that span more than 2 lines. Some examples:
  * “Given the expected data sparsity in this setting, beyond a particular user’s data, such personalized language systems will likely also rely on historical data from other (similar) users to learn how to learn from a small set of new user feedback (see Figure 2).” Could be simplified/broken (by an LLM): “These personalized language systems will likely use more than just one user's data due to the expected data sparsity in this setting. They will also depend on historical data from other similar users. This helps them learn effectively from a small amount of new user feedback (see Figure 2 for more details).”
  * “We do not claim our simulated personal preference models provide a high-fidelity depiction of human behavior, but rather offer a challenging simulation environment that provides the empirical foundation for methodological innovation in capturing the complex array of human preferences that arise in practice.” Could be made easier to read (by an LLM): “We don't claim that our simulated personal preference models perfectly mimic human behavior. Instead, they offer a challenging simulation that provides a basis for developing new methods. This helps in better capturing the complex range of human preferences encountered in real life.”
  * “While human evaluation like that of Kirk et al. (2024) is a gold standard, wherein fine-grained preference feedback is gathered from a representative sample of diverse and multicultural participants, it is impractical or even impossible to get this feedback throughout the methodology development cycle, meaning that synthetic personal preference models will ultimately be needed.” I had to read this one slowly a couple of times…
  * Line 354: “Two first-order problems…” can be losslessly simplified to “Two problems…”.
* Line 254: choosing only 500 personas may be too little if the goal is to achieve heterogeneity, especially since 1000 users are sampled for PersonalLLM. Can you please include results with 1000 personas? It may actually be interesting to see how the results change when increasing the sample size for both persona and PersonalLLM.
* Line 257: “we can see that the top response receives a majority user vote for only about half of the prompts, while that figure is closer to 90% for the persona prompting baseline.” Sorry, I could not read that from the figure, can you please explain how the results show this?
Also in line 258: “Also, for roughly 60% of prompts, at least 5 different answers are chosen as the best by at least 1 under our set of personas; for LLM persona prompting, it is roughly 30%.” Please explain.
* Line 274: “With respect to changes across the left 3 columns, we can observe that as α increases, preferences become more uniform. However, if α is set too low, user preferences cluster very tightly around the base reward models; we observe this behavior for α = 0.01.” — looking at the figure, it actually seems like there is not much difference between the first 3 columns. Is there a better way to show this difference?
* Line 294: “In Figure 5 (right), we compare the entropy in the population preferences over the responses to a given prompt based on keywords, comparing words we would expect to inspire heterogeneity (e.g., imagine, opinion, poem) to prompts beginning with “who”, “when”, and “where”, which evoke more objective answers.” This was not clear to me, maybe add a formal definition and/or an equation for the entropy? Also, how do standard personas compare to the proposed approach in this task?
* In Section 4.2, is it mentioned how response (and prompt) embeddings are computed?

Minor/typos:
* Line 32: Christiano et al., 2017, not 2023
* In Figure 6 (left), the dashed line is missing from the legend. I am guessing this is the zero-shot performance.
* The large-scale dataset could be useful for LLM development.
* The alternative to persona-based simulated users seems novel.
This paper presents a new dataset of simulated preferences. The data consists of 10K prompts X 8 responses from different LLMs for each prompt X 10 rewards from different reward models. 1000 simulated users are sampled, where each user’s preferences are defined by a weighted sum of rewards (the weights are sampled from a Dirichlet distribution). The data is then used in in-context learning (ICL) for improving the LLM responses w.r.t. the user’s preferences.

Personalization is achieved by ICL, adding examples of good/bad responses according to the weighted reward. The results (Figure 6 left) show that using ICL with historical preferences can improve performance compared to zero-shot.

Learning across users is proposed, retrieving other users with similar preferences from a set of simulated users, and using their preferences for ICL. The results (Figure 6 right) show a small improvement when using both positive and negative preferences compared to ICL using only the user’s history.
* It is stated in the paper that the goal is not to match preferences of a distribution of real users, but rather to generate diverse preferences that are more heterogeneous/diverse. I think that this requires more justification since random preferences would give even higher diversity but may not be useful.
* Clarity/readability could be improved (see detailed questions).
@inproceedings{
anonymous2024personalllm,
title={Personal{LLM}: Tailoring {LLM}s to Individual Preferences},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=2R7498e2Tx},
note={under review}
}
As LLMs become capable of complex tasks, there is growing potential for personalized interactions tailored to the subtle and idiosyncratic preferences of the user. We present a public benchmark, PersonalLLM, focusing on adapting LLMs to provide maximal benefits for a particular user. Departing from existing alignment benchmarks that implicitly assume uniform preferences, we curate open-ended prompts paired with many high-quality answers over which users would be expected to display heterogeneous latent preferences. Instead of persona prompting LLMs based on high-level attributes (e.g., user race or response length), which yields homogeneous preferences relative to humans, we develop a method that can simulate a large user base with diverse preferences from a set of pre-trained reward models. Our dataset and generated personalities offer an innovative testbed for developing personalization algorithms that grapple with continual data sparsity---few relevant feedback from the particular user---by leveraging historical data from other (similar) users. We explore basic in-context learning and meta-learning baselines to illustrate the utility of PersonalLLM and highlight the need for future methodological development.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Personalization LLM Alignment benchmark dataset reinforcement learning from human feedback language models RLHF preferences
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/ef95d073d0933e544391f42678b1a19474978fa9.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/c773caff40399be446d5213e0533de5c3f1e8c99.zip
PersonalLLM: Tailoring LLMs to Individual Preferences
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission