Online Personalizing White-box LLMs Generation with Neural Bandits
Zekai Chen Weeden Daniel Po-yu Chen Francois Buet-Golfouse

Cybersecurity Assessment of the Polar Bluetooth Low Energy Heart-rate Sensor
    S. Soderi1
    
============================================================================




The advent of personalized content generation by LLMs presents a novel challenge: how to efficiently adapt

text to meet individual preferences without the unsustainable demand of creating a unique model for each user. This study introduces an innovative online method that employs neural bandit algorithms to dynamically optimize soft instruction embeddings based on user feedback, enhancing the personalization of open-ended text generation by white-box LLMs. Through rigorous experimentation on various tasks, we demonstrate significant performance improvements over baseline strategies. NeuralTS, in particular, leads to substantial enhancements in personalized news headline generation, achieving up to a 62.9% improvement in terms of best ROUGE scores and up to 2.76% increase in LLM-agent evaluation against the baseline. 





§ INTRODUCTION












In recent years, the advancements in large language models (LLMs) have been remarkable <cit.>, with these models demonstrating an unparalleled ability to understand and generate text across a wide spectrum of tasks <cit.>. This capability has 

revolutionized the way we interact with machine-generated content 

and 
opened up new avenues for personalized text generation <cit.>. 

Personalization in text generation is of paramount importance 

to ensure
user engagement and satisfaction <cit.>

across a range of in applications such as composing tweets, or generating news articles and financial reports, or in more personalized settings like business communications and creative writing <cit.>.




However, the prospect of developing a unique LLM for each user presents 

challenges, including the prohibitive resource requirements <cit.>, data privacy concerns <cit.>, and the scarcity of personalized data <cit.>. These obstacles necessitate an alternative strategy that is both practical and flexible. A promising solution lies in adopting lightweight models capable of online learning, which can dynamically adjust their output based on continuous user feedback <cit.>. Such an approach not only circumvents the need for a bespoke model for each user but also encourages 

alignment of the 
generated content 

to individual preferences over time. Importantly, this adaptive process is poised to unlock long-term rewards stemming from personalization, encompassing not just explicit preferences expressed by users but also responding to favorable actions <cit.>. 

Despite these benefits, the ultimate effectiveness of LLMs hinges on the quality of the given instructions <cit.>. Previous efforts have focused on gradient-based strategies <cit.> for automated instruction optimization, the applicability is limited to less advanced public models, leaving out many advanced yet proprietary models. With the emergence of more advanced open models such as Mistral-7B <cit.>, Llama-70B <cit.>, and Mixtral-8x7B <cit.>, which offer transparency and have reported performance that even surpasses that of ChatGPT-3.5[https://chat.openai.com/], there is a renewed focus on leveraging these models for direct optimization. 



In this study, we introduce a novel online method for enhancing the personalization of open-ended text generation with white-box LLMs. Considering that capturing the nuances of persona in natural language instructions is challenging, we aim to directly optimize the soft token embeddings <cit.>, representing the contextual factors through user feedback by utilizing neural bandit algorithms <cit.>. This method not only promises to refine the personalization process of text generation but also contributes to the broader application of adaptive algorithms in creating content that closely reflects individual user preferences.



§ PERONALIZATION WITH NEURAL BANDITS





Neural Bandits <cit.> integrate the adaptive exploration of bandit algorithms <cit.> with neural networks' superior ability to predict rewards under uncertainty. By leveraging past interactions to balance the trade-off between exploring new actions and exploiting known ones, these algorithms can accurately predict and enhance personalized outcomes. Therefore, we adopt NeuralUCB <cit.> and NeuralTS <cit.> in our framework to comprehensively evaluate how Neural Bandits benefit LLMs' generation in an online fashion. 




This section explores the application of Neural Bandits to white-box large language models (LLMs) as a strategy for directly refining soft prompts (aka. contextual embeddings) to overcome the inability of natural language instructions to fully express nuances of persona.
The process involves a white-box LLM, f, which takes a soft prompt z, an instruction, ρ, and a test input, x, to produce an output sentence, ŷ = f(z, ρ, x). A soft prompt z is a continuous vector representing preference token embeddings, which is used alongside the default instruction, ρ, as input to f. Given a specific user u at time t, we have a specific input x_i, the goal is to find an optimal soft prompt token, z^*, that maximizes the following objective function based on user's feedback s(ŷ_i) (see Figure <ref> for illustration):

    z_t^* = max_z ∈ ZNeuralBandits_t(z, s)

With NeuralUCB, it involves computing the acquisition value for each candidate soft prompt and selecting the one that maximizes this value, based on the model's current parameters and the uncertainty associated with each prompt:

    z_t+1 = max_z ∈ Z(μ(g(z); θ_t) + ν_t σ_t(g(z); θ_t) ),

Here, μ(g(z); θ_t) represents the network's predicted value for the soft prompt z at iteration t, while σ_t(g(z); θ_t) quantifies the uncertainty of the prediction. The parameter ν_t controls the trade-off between exploration and exploitation, influencing the algorithm's preference for exploring less certain prompts versus exploiting prompts with higher predicted values.

When using NeuralTS for updating z, the process involves sampling from a predictive distribution to select the next soft prompt. Unlike NeuralUCB, which directly uses a deterministic acquisition function, NeuralTS generates a sample for each candidate z from a distribution modeled by the neural network. The update can be represented as follows:


    z_t+1 = max_z ∈ Zr̃_z,t


    r̃_z,t∼𝒩(μ̃(g(z); θ_t), ν_t σ̃_t(g(z); θ_t))


Where, μ̃(g(z); θ_t) and σ̃_t(g(z); θ_t) 
represent the estimated mean and standard deviation respectively for the soft prompt z, and r̃_z,t is a sampled reward from the predictive distribution for z at iteration t. The parameter ν_t again balances exploration and exploitation, but in the context of NeuralTS, the exploration is informed by the stochasticity introduced through sampling, encouraging diversity in the selection of z based on both prediction and uncertainty.





NeuralUCB uses an upper confidence bound to balance these aspects deterministically, offering robust performance in environments where a clear quantification of uncertainty benefits decision-making. NeuralTS, on the other hand, employs a Thompson sampling <cit.> approach, introducing stochasticity in the selection process, which can lead to more diverse exploration. Therefore, we tested both approaches as different kernels given the actual performance hinges on the problem's nature, the desired balance between exploration and exploitation, and the computational resources available.




§ EXPERIMENTS ON LAMP





 §.§ Personalized Generations

The LaMP <cit.> dataset, or Language Model Personalization, is a benchmark designed for training and evaluating large language models (LLMs) to produce personalized outputs. It aims to assess the efficacy of LLMs in generating responses tailored to individual user profiles. For example, in personalized news headline generation task, the writing of journalist exhibits unique stylistic characteristics shaped by both individual and societal influences <cit.>. This scenario serves as an excellent opportunity for exploring personalized generation.



In this study, we evaluate our framework on three open-ended generation tasks which are Personalized News Headline Generation, Personalized Scholarly Title Generation, and Personalized Tweet Paraphrasing. 







 §.§ Online Simulation using LaMP




We simulate a realistic online setting where individual users, such as those represented by distinct profiles in the LaMP dataset, continually receive outputs from LLMs in response to predetermined instructions. We then take ROUGE scores <cit.>, which measure the correspondence between the model's output and the user's ideal or "golden" response, as their online feedback. Additionally, we monitor the assessments from a black-box LLM, which evaluates the appropriateness of the generated content for a desired persona based on their profiles (e.g., using LLMs to automatically rate the generation based on the consistency with users' stylistics). An ideal result would exhibit both improved ROUGE metrics and LLM ratings.



As introduced in Section <ref>, we employ Mistral-7B-Instruct-v0.2[https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2] as our default white-box LLM. By simulating this online interaction scenario, our aim is to closely mimic the dynamic and personalized experience users encounter in real-world applications of language models. Through this experiment, we seek to evaluate the effectiveness of Neural Bandits in responding to the nuanced persona of different users/profiles, thereby informing future improvements and adaptations in personalized generation.



 §.§ Results Analysis



For the task of Personalized News Headline Generation, from Table <ref> we observed a significant performance leap with NeuralTS, achieving an Average ROUGE-1/L score of 0.228 ± 0.088, marking a 62.9% improvement over the random baseline (0.140 ± 0.076). NeuralUCB also outperformed the baseline, albeit with a slightly lower score of 0.203 ± 0.072. In Personalized Scholarly Title Generation, NeuralUCB exhibited the highest increase, with a score of 0.345 ± 0.116, closely followed by NeuralTS (0.341 ± 0.119), both surpassing the baseline (0.225 ± 0.127) by approximately 53.3%. Lastly, the Personalized Tweet Paraphrasing task showed NeuralTS slightly outperforming NeuralUCB with a score of 0.466 ± 0.120 against 0.459 ± 0.114, over a baseline of 0.346 ± 0.110, translating to a 34.7% improvement. Figure <ref> also summarises these results. At the same time, as shown in Figure <ref>, we observe increase over the LLM evaluation scores, with NeuralUCB achieving an average improvement of 2.8% vs. ∼1% for NeuralTS. These findings underscore the potential of Neural Bandit algorithms in enhancing content personalization across various text generation tasks. 



However, it's important to acknowledge certain limitations inherent in LLM evaluations. LLMs tend to exhibit a positivity bias, often returning higher scores for the generated content. This bias can potentially skew the evaluation results and may not fully capture nuances in personalization or consistency. Also, while ROUGE provides a useful automated proxy for assessing the quality of personalized content, it has well-known limitations in fully capturing subjective preferences or nuanced stylistic tendencies. It may not align well with human judgments, especially for highly creative and personalized text. Thus, while the gains demonstrated in both ROUGE and LLM evaluation provide encouraging evidence, human evaluations are imperative for comprehensively assessing improvements in adapting to individual user profiles. 







§ CONCLUSION

In conclusion, this study presents a novel framework for personalized content generation using large language models (LLMs) through the application of neural bandit algorithms. By dynamically optimizing soft instruction embeddings through online feedback, our method demonstrates improvements in personalizing open-ended text generation tasks as scored by both ROUGE scores and an LLM evaluator. Further research is needed to confirm that these results are consistent with human-evaluated suitability.






§ LIMITATIONS

We discuss some limitations this work may present in this section. The evaluation in this study is limited to only three generation tasks from the LaMP benchmark. Testing on a more diverse and extensive set of personalized taks would strengthen the claims around the general applicability of the method. The simulation of an online environment with user profiles from the LaMP dataset may not fully capture the nuances of real-world user interactions and feedback. Evaluating with actual human subjects over longer time horizons could reveal additional challenges. Additionally, ROUGE score, is used to assess generation quality. Incorporating other metrics and human evaluations could provide a more holistic view of the improvements achieved. As of now, only two neural bandit algorithms, NeuralUCB and NeuralTS, are investigated. Comparing to a wider range of adaptive optimization algorithms could provide deeper insights into the most suitable techniques.



§ ETHICS CONSIDERATION

In real-world, collecting personalized data like past personal data raises privacy concerns around data usage and consent. In our study, we use LaMP which is a public dataset with user consent. To some extent, tailoring content to align with a user's preferences risks creating isolated filter bubbles that entrench viewpoints. This is also an important reason that we investigate bandit algorithms to better balance exploration against exploitation and serve as mitigation strategies which exposes users to diverse perspectives. One potential risk is that the capability to generate highly persuasive personalized text could be misused for deceptive purposes. Safeguards against generating harmful, unethical or untruthful content need to be in place. Like any powerful technology, personalized generation abilities could be co-opted for nefarious ends by malicious actors. Responsible disclosure and governance practices are critical. In summary, the ethical dimensions span privacy, security, fairness, transparency, accountability and the broader social impacts of deploying personalized generative models. A principled, human-centric approach that places ethics at the foundation of research and development is imperative as these capabilities continue to advance.

























acl_natbib





§ APPENDIX





 §.§ Hyperparameters for NeuralBandits

Following InstructZero <cit.> and Instinct <cit.>, the reported results were based on hyperparameters optimized as:

- Intrinsic dimension - The soft prompt z often has very high dimensionality (e.g. d=4096× N_z for Mistral-7B), making it challenging to directly optimize via Bayesian optimization. To address this, InstructZero <cit.> employs random projection as an effective dimensionality reduction technique. Specifically, a random projection matrix 𝒜∈ℛ^d× d' is used, where d'≪ d is the reduced intrinsic dimension. Given a d'-dimensional vector z' and the projection matrix A, the original high-dimensional prompt z is computed as z = Az'. By optimizing over z' instead of z, the dimensionality of the optimization problem is reduced from d to the much lower d'. d' is known as the intrinsic dimension in this case. We set it as 100 by default usage. 

- Number of soft tokens - N_z represents the length of contextual embeddings or soft tokens that are utilized to concatenate with initial instructions fed into LLMs. Ideally, we would like to balance off this parameter so it's able to capture the persona nuances while also not causing extra compute redundancy. This number is set to 5 in our experiments. 

- λ - Controls the strength of the prior in the NeuralUCB acquisition function. Set to 0.1 after grid search.

- ν_t - balances exploration versus exploitation in Neural Bandits. Fixed at 0.1 based on best validation performance.

- Total iterations - total update iterations for neural bandits, set to 165 in practice.

- Hidden dimensions - We used an MLP on top of the LLM embeddings with 100 hidden units. This architecture achieved the best validation results.

- Local iterations - represents how many learning steps for neural network to retrain every update iteration. In order to prevent overfitting but also avoid underfitting, we set this number as 40 in practice.

- Learning rate - The MLP was trained with an AdamW optimizer <cit.> using a learning rate of 3e^-4. 

- Projection matrix - Random projections for dimensionality reduction were generated by sampling from a Uniform(-1, 1) distribution.

For simplicity, we randomly select 100 profiles from each task for experiments and all the reported results are average with standard deviation using the same seed 42. All experiments are conducted on 4 Nvidia A10g GPUs. 



 §.§ Prompts for LLM Evaluation

For LLM-agent evaluation, we design prompts for black-box LLM that evaluates the white-box LLMs generation in a more comprehensive way. Please refer to Figure <ref> for the details.


