Preference Discerning in Generative Sequential Recommendation
Generative Retrieval;Sequential Recommendation;Preference Discerning;LLM
Yes
No ethics review needed.
Please refer to weaknesses
-	This paper uses four diverse datasets to ensure the generalizability and reliability of the experimental results.

-	The proposed model, Mender, is evaluated across multiple dimensions such as preference-based recommendation, sentiment following, fine-grained and coarse-grained steering, and history consolidation. The results show that Mender significantly outperforms existing state-of-the-art methods, particularly in preference guidance and sentiment following, demonstrating its robustness and effectiveness.
This paper aims to enhance personalized recommendations by explicitly incorporating user preferences and historical interactions. The proposed method Mender uses a pre-trained language model to generate user preferences from comments and integrates these preferences with historical data using cross-attention mechanisms. Experimental results show that Mender outperforms existing state-of-the-art methods.
-	The methodology section should be reorganized to provide a detailed explanation of the preference generation process. Mathematical formulations are expected to be included for explicit understanding, and pseudo-code is recommended to enhance clarity and reproducibility.
-	It is kindly recommended to add further discussion about how does the benchmark generation benefit personalization modeling.
Yes
No ethics review needed.
1. What is the major technical contribution of MENDER? 
2. Are there some direct and objective evaluation methods to check the effectiveness of preference-discerning results? 
3. What is the motivation for using the generative recommendation pipeline? 
4. Is MENDER a efficient method compared with traditional sequential recommendation baselines as SASRec?
- Novel Benchmark. The authors suggest a new benchmark to evaluate the personalization ability of the user preference description. 
- Abundant Results. Extensive experiments are conducted. 
- Credible Reproduction. Reproduction details are available in Appendix.
This paper proposes MENDER, aiming to enhance the personalization of sequential recommendation models. Specifically, the authors first design a preference discerning paradigm based on zero-shot LLMs. With the obtained user preference, the authors construct the generative recommendation framework based on RQ-VAE. Extensive experimental results are provided to show its effectiveness.
- Limited Technical Contribution. Overall, the framework is constructed based on existing works. The proposed method, MENDER, mainly consists of two modules, RQ-VAE and feed-forward components (Emb or Tok). Other researchers have suggested both modules, which may indicate the limited technical contribution of this work. 
- Lack of the Cross-validation on Benchmark. The suggested "holistic" benchmark is subjective and not double-checked by objective ground truth. The overall performance only reflects the indirect effectiveness of additional preference summarization, while the success of preference discerning should be further validated. 
- Inadequate Motivation. The motivation for enhancing the personalization and applying the generative recommendation is not supported. How do authors define "personalization" and examine "personalization"? Why do authors only construct the generative model? Can we integrate MENDER with discriminative models? 
- Unknown Efficiency. The efficiency of the proposed framework has not been tested.

Minor problems:
- The last block of Table 2 is in the wrong format. 
- Code is not available.
Yes
No ethics review needed.
Why do you think the five factors in your "evaluation benchmark " are equally important and are the only concerns by sequential recommendation systems?

What is the difference between your proposed preference generation framework with existing ones (as listed in "weakness" section).
1. The authors would like to propose a thorough evaluation framework (the authors claim it to be a "benchmark") for sequential recommendation system, which is a nice direction.

2. The experimental results are credible and show improvement compared with some existing baselines.
The submission focuses on sequential recommendation technique. The main contributions lie that (1) the authors proposed a LLM-based user preference generation method based on user generated reviews and (2) they propose an evaluation framework that contains five different aspects that should be taken into consideration by sequential recommendation systems.
1. The novelty is rather limited. Extracting user preference information from their reviews is not novel. However, these existing works are not mentioned or compared by the authors. Some examples include:

User-LLM: Efficient LLM Contextualization with User Embeddings, https://arxiv.org/abs/2402.13598  (The work investigates how to capture latent user behaviors into preferences)
Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction, https://arxiv.org/abs/2305.06474 (This work investigates how LLMs comprehend user preferences and compares the performances of different LLMs in this issue.)
Review-driven Personalized Preference Reasoning with Large Language Models for Recommendation, https://arxiv.org/abs/2408.06276 (this work proposes to extract subjective preferences from raw reviews, which is a key contribution the authors claim)

2. The proposed evaluation benchmark is rather straightforward and not so reasonable. This kind of framework should be validated by either product managers or large scale user studies. The radar chart indicates that these five dimensions are equally important, which is also not validated by any evidence. If the authors would like to propose such a framework, I would suggest compare the proposed one with actual user experiences through practical user studies.
Yes
No ethics review needed.
1. Given the results of Mender$_{Tok}$-Pos-Neg in Figure 5, achieving the best sentiment following results does not necessarily ensure the model's proficiency in other dimensions. Does the pursuit of high performance in sentiment following adversely affect the model's overall capabilities?
2. Why is the time factor not considered, given that user preferences can change significantly over time, especially considering that the time span of the datasets can be decades? What's the implications of not considering the time factor in the current model?
3. In Section 4,  Equation 1 already takes every item in the sequence except the last item $i_{T_{u}}$ into account, then what's the meaning of "repeating this generation process for each item in $s_{u}$" in line 203?
4. In Section 4.1, line 237, why the steering ability can be achieved by creating new sequences? Is there a reference can prove this? In Appendix D.2, line 1276, a figure reference is missing.
5. Still in Section 4.1, line 241, how is $p_{1}$ and $p_{2}$ achieved? What's the point of combining them with new sequences?
6. Still in Section 4.1, line 243, since $\hat{i}_t$ represents a distinct item, why its sequence combines with $p_1$, which represents the preference of a similar item?
7. Why the NDCG results of sentiment following are not provided in Table 2 and other tables in the Appendix?
1. This paper identifies a critical issue in sequential recommendation: the failure to explicitly capture and utilize user preferences, prompting the introduction of a new task: preference discerning.
2. It proposes a novel benchmark comprising five key dimensions to evaluate the preference discerning abilities of existing models.
3. The paper enhances the RQ-VAE framework by directly representing both user interaction history and preferences, while also introducing two variants that encode inputs in different ways.
4. Extensive experiments are conducted, accompanied by detailed analysis to validate the findings.
This paper introduces a new benchmark and proposes a novel method called Mender to evaluate and enhance the preference discerning capabilities of sequential recommendation systems. The benchmark assesses models across five dimensions, focusing on their capacity to extract and utilize user preferences from datasets. Recognizing that existing methods lack key capabilities of preference discerning, the authors propose Mender, a multimodal generative retrieval approach which effectively extracts user preferences and achieves state-of-the-art performance on the proposed benchmark. Experimental results further demonstrate the effectiveness of the method.
1. No NDCG results for sentiment following are reported in Table 2 or the Appendix. I assume the results are close to zero, suggesting that a logarithmic scale should be used to better analyze the discrepancies between the models. 
2. The time factor is not considered, given that user preferences can change significantly over time, especially considering that the time span of the datasets can be decades. Simply limiting the user sequence to the 20 most recent items does not fully eliminate time bias. Instead, the time interval of user-item interactions should be restricted during sampling to better capture user preferences.
3. The methodology outlined in Section 4 is unclear, with similar issues arising in Section 4.1 on Fine-Grained & Coarse-Grained Steering, where the concepts are not adequately explained. In Equation 1, the entire sequence is considered, while the subsequent statement describes repeating the process for each item in the sequence, leading to ambiguity. Furthermore, in Fine-Grained & Coarse-Grained Steering, there is no reference provided to justify the validity of this approach. The sequence processing in this section also lacks rationality, as it combines a distinct item $\hat{i}_t$ with $p_1$, which represents the preference of a similar item. 
4. Experiments with only three baselines is not convincing enough, new baselines should be added, including https://arxiv.org/abs/2311.09049 (Zheng, Bowen, et al. "Adapting large language models by integrating collaborative semantics for recommendation." ICDE 2024). This paper also employs RQ-VAE, in which LLM-encoded text embedding of the item is utilized as input.
Yes
No ethics review needed.
Please refer to weaknesses
1. The motivation of this work, leveraging user preference in recommender system, is good.
2. The author conducts extensive experiments.
This paper first introduces a new benchmark to evaluate the model's ability to capture user preference. Then Mender is proposed to integrate LLM-generated user preferences to enhance the generative recommender system. Experiment results on its proposed benchmark show improvement on the proposed method.
Regarding Benchmark Design:

1. While preference-based recommendation is undoubtedly a core aspect, the practical value of the tasks such as Sentiment Following, Fine-Grained & Coarse-Grained Steering, and History Consolidation is questionable. This raises concerns about the overall contribution of the benchmark.
2. The Fine-Grained & Coarse-Grained Steering task is confusing. The paper states, “we associate these two items with different user preferences, denoted as p1 and p2, respectively,” but the relationship between p1, p2, and similar or distinct items is unclear. How are “different user preferences” determined? Additionally, in the new sequences created, why is p1 added to the sequence of very distinct items while p2 is added to the ground truth item sequence? This contradicts the earlier association of p1 and p2 with similar and distinct items, respectively. What role does the similar item play?
3. The design of the Sentiment Following task does not adequately reflect the model’s ability to follow sentiment. The description is also unclear, and I suggest the authors reorganize this section.
4. The practical value of History Consolidation is questionable, and its evaluation metric seems unnecessary. Why not train the model directly using the five preferences? The paper claims to “infer which preference is most relevant for predicting,” but there is no experimental evidence demonstrating this capability. In fact, the performance with multiple preferences is even worse than with a single preference.
5. The experimental discussion on each task, particularly Sentiment Following and History Consolidation, is insufficient.

Regarding Presentation:
1. Missing reference on line 1275.
2. Typo on line 1167: "tr iggered" should be "triggered."
3. Typo on line 401: "48.3.4%" should be corrected.
4. Method names are displayed incorrectly in lines 403-406.
5. In Table 2, the performance drop for History Consolidation on the Steam dataset seems miscalculated. The relative decline should be based on the better-performing Mender-emb, not Mender-Tok.
6. The section titled "PREFERENCE DISCERNING" in part three should likely be part of the Methodology (Section 4.2). It is unclear why this is presented as a separate section.

Regarding Experiments:
1. The selection of baselines is insufficient, with only three included. One of these, TIGER, is an ID-based model that does not leverage preference, making the comparison unfair. The two VocabExt variants either introduce a gap between randomly initialized item embeddings and semantic information, or they lack pre-trained preference understanding, making them variants of TIGER rather than fair comparisons. The authors should consider two sets of baselines: (1) preference-based recommendation models and (2) advanced TIGER variants, such as LETTER, LC-Rec.
2. The statement in line 282, “Mender-Emb allows pre-computing item and preference embeddings, resulting in improved training efficacy,” conflicts with the experimental results, as Mender-Emb consistently underperforms compared to Mender-Tok in Table 2.
3. Although the benchmark is a key contribution of the paper, there is insufficient discussion of most tasks in the experimental section, especially History Consolidation and Sentiment Following.
The lower performance of History Consolidation compared to Recommendation raises questions about the usefulness of combining five preferences versus a single preference. This casts doubt on both the validity of the preference design and the method’s ability to effectively leverage preferences. Additionally, the abnormal results on the Steam dataset lack sufficient discussion and explanation.
We propose a new paradigm called preference discerning along with a benchmark and a new baseline and evaluate capabilities of state-of-the-art generative retrieval methods
@inproceedings{
anonymous2024preference,
title={Preference Discerning in Generative Sequential Recommendation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=3ZDMQGQgkE},
note={under review}
}
Sequential recommendation systems aim to provide personalized recommendations for users based on their interaction history. To achieve this, they often incorporate auxiliary information, such as textual descriptions of items and auxiliary tasks, like predicting user preferences and intent. Despite numerous efforts to enhance these models, they still suffer from limited personalization. To address this issue, we propose a new paradigm, which we term *preference discerning*. In *preference discerning*, we explicitly condition a generative sequential recommendation system on user preferences within its context. The user preferences are generated by large language models (LLMs) based on user reviews. To evaluate *preference discerning* capabilities of sequential recommendation systems, we introduce a novel benchmark that provides a holistic evaluation across various scenarios, including preference steering and sentiment following. We assess current state-of-the-art methods using our benchmark and show that they struggle to accurately discern user preferences. Therefore, we propose a new method named Mender (**M**ultimodal prefer**en**ce **d**iscern**er**), which improves upon existing methods and achieves state-of-the-art performance on our benchmark. Our results show that Mender can be effectively guided by human preferences, paving the way toward more personalized sequential recommendation systems. We will open-source the code and benchmarks upon publication.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Generative Retrieval Sequential Recommendation Preference Discerning LLM
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/0dd9882c0a8967228ca276ee46b44b0ce6f352ff.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Preference Discerning in Generative Sequential Recommendation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission