abstract: Prompt engineering relevance research has seen a notable surge in recent years, primarily driven by advancements in pre-trained language models and large language models. However, a critical issue has been identified within this domain: the inadequate of sensitivity and robustness of these models towards Prompt Templates, particularly in lesser-studied languages such as Japanese. This paper explores this issue through a comprehensive evaluation of several representative Large Language Models (LLMs) and a widely-utilized pre-trained model(PLM). These models are scrutinized using a benchmark dataset in Japanese, with the aim to assess and analyze the performance of the current multilingual models in this context. Our experimental results reveal startling discrepancies. A simple modification in the sentence structure of the Prompt Template led to a drastic drop in the accuracy of GPT-4 from 49.21 to 25.44. This observation underscores the fact that even the highly performance GPT-4 model encounters significant stability issues when dealing with diverse Japanese prompt templates, rendering the consistency of the model's output results questionable. In light of these findings, we conclude by proposing potential research trajectories to further enhance the development and performance of Large Language Models in their current stage.
author: Chengguang Gan Tatsunori Mori
Yokohama National University, Japan
`email@example.com, firstname.lastname@example.com`
bibliography: custom.bib
title: Sensitivity and Robustness of Large Language Models to Prompt Template in Japanese Text Classification Tasks

# Introduction

With the advent of the transformer architecture, language models have transitioned into the pre-training epoch. The concept of prompt-based learning emerged as a method for adapting downstream text classification tasks into fill-in-the-blank problems, thereby facilitating compatibility with the input requirements of these tasks. This approach optimizes the alignment between the downstream task's input and the pre-training objective of BERT models, specifically the Masked Language Model. Consequently, the model exhibits accelerated learning capabilities, even when exposed to a limited number of samples.

The manual methodology for designing prompts presents several limitations, as illustrated in Figure 1 using a sentiment classification task. Even though various prompts may appear semantically similar from a human perspective, they often yield considerably different results when implemented in the same model. Furthermore, the manual design process does not provide a definitive way to ascertain the optimal prompt template for a given task and model. Remarkably, a single prompt with ostensibly identical meaning can exhibit a wide array of performances. However, it is not feasible to experiment with all variations to identify the superior template. In subsequent research, Automated Prompt Template methods like AutoPrompt and Prefix-Tuning have been introduced. However, their applicability in the context of Large Language Models (LLMs) remains limited, hence, they will not be explored in this paper.

The introduction of ChatGPT has been met with considerable admiration due to its impressive performance. Nevertheless, questions persist concerning the model's sensitivity and robustness towards different prompt templates. Has ChatGPT effectively addressed these issues or at least mitigated them?

In addition to the extensive body of evaluation literature demonstrating the robustness of ChatGPT in major languages such as English and Chinese, there is a conspicuous lack of similar studies focused on less dominant languages. In this study, we focus on Japanese, one of the minor languages. Japanese is characterized by distinctive rules and grammatical constructs that set it apart from other languages. These idiosyncrasies potentially complicate the construction of prompt templates, thereby posing unique challenges to the robustness and sensitivity of multilingual LLMs. Consequently, a key question this research aims to answer is: how well can existing multilingual LLMs accommodate the prompt templates unique to diverse languages such as Japanese?

To explore this question, we have undertaken a study which involves conducting experiments to assess the performance of existing LLMs for Japanese, a language that is less commonly modeled. This paper principally delivers the following key contributions: 1. A comprehensive evaluation of the robustness and sensitivity of LLMs is conducted concerning the influence of prompt templates on a Japanese benchmark dataset. The empirical findings disclose both the current advantages and shortcomings of LLMs. 2. The performance of LLMs is scrutinized through the application of an array of prompt templates, which indicates that the choice of specific words or phrases in Japanese prompt templates significantly impacts the LLMs' sensitivity. This insight, to a substantial degree, offers valuable guidance towards optimizing the model's training methodologies and corpus construction. 3. The experiments expose a substantial stability issue in LLMs when handling Japanese prompt templates of analogous meanings. This discovery subsequently introduces a novel avenue for enhancing the efficacy of future LLMs in Japanese.

# Related Work

In this section, we elucidate upon existing studies that examine the performance of language models when faced with various prompts.

In the field of PLM. Examines the robustness of natural language models to domain shifts using prompt tuning and prefix tuning methods. Significant differences were found in domain robustness patterns between the two methods on T5 and GPT-2 models. The authors call for future research to explore the causes of these variations. In the realm of LLMs, prevailing research is predominantly centered on ChatGPT. One primary area of focus within this domain is the exploration of its application in translation tasks. Evaluates ChatGPT's ability to perform machine translation tasks. The paper shows that ChatGPT performs competitively with commercial translation services on high-resource languages, but struggles with low-resource or distant languages. However, a pivot prompting strategy improves performance significantly.

The comprehensive study by , evaluates the robustness of GPT-3.5 on 21 datasets across 9 NLU tasks using TextFlint's text transformations. Findings show GPT-3.5 outperforms fine-tuned models in some tasks but experiences significant performance drops in others. Specific challenges include robustness instability, prompt sensitivity, and number sensitivity. The study also highlights variations in performance between different GPT models and the need for consistency in task labels and label types. These insights are crucial for understanding limitations and guiding future research to improve AI model performance and generalization abilities. However, limits its scope to the examination of mainstream Chinese and English languages. It does not delve into a comprehensive evaluation and analysis of other less predominant languages.

There is also some work using LLMs, with Japanese as the research objectives. For example, testing the performance of GPT-4 and ChatGPT in Japanese Medical Licensing Examinations. Furthermore, using the GPT-3 model and multilingual prompt method for Japanese natural language tasks, above baseline results were achieved.

While previous experiments have provided valuable insights, they have not delved into the sensitivity and robustness of LLMs pertaining to the specific linguistic nuances of the Japanese language. Our primary objective is to **investigate the impact of subtle variations of prompt template on**

```latex
\begin{table*}[!t]\centering
\setlength{\tabcolsep}{9pt}

\begin{tabular}{lcc}
\toprule[2pt]
  \textbf{Dataset} & \textbf{Category} & \textbf{Total number of categories}\\
\midrule
\textbf{MARC-ja} & positive,negative & 2\\
\textbf{JNLI} & contradiction,neutral,entailment & 3 \\
 % & ,entailment &  \\
\textbf{JSTS} & 0,1,2,3,4,5 & 6  \\
\bottomrule[2pt]
\end{tabular}
\caption{\label{table1}
Details of the Datasets (Total refers to the number of categories)}
\end{table*}
```

**model performance, especially when the meanings of the prompt templates bear a close resemblance**.

# Experiment Setup

This section commences with an introduction to the general parameters of the experiment. Subsequently, we delve into the three pivotal aspects of the experiment specifically the dataset, the model, and the design of the prompt template in a more comprehensive manner.

The experiments are initially established as zero-shot. This is primarily due to the fact that, in most real-world implementations of LLMs, the responses are provided directly to the users' inquiries, without any domain-specific fine-tuning. Consequently, our experiment will exclude the utilization of few-shot or In-context Learning methodologies. The model will not be provided with a set of examples to learn from. Adopting a zero-shot approach also circumvents the issue of bias that could arise when selecting samples for a few-shot experiment.

In regard to the model's output settings, the maximum number of new tokens is limited to 10. Additional parameters are not deliberately defined at this stage, as our aim is to control the variables of the experiment as rigorously as possible. This is done to safeguard the fairness and integrity of the overall comparative study. Furthermore, due to constraints in time and resources, we have selected a test set size of 1000. The random seed has been set at 42, which is arbitrarily selected from the dataset.

## Dataset

In our dataset selection, we employed the Japanese General Language Understanding Evaluation (JGLUE) benchmark, which encompasses five distinct datasets: text classification, sentence pair classification, and QA tasks. We focused on three specific datasets predominantly geared towards text classification and sentence pair classification tasks, given the fundamental importance of sentence classification within Natural Language Processing (NLP) as an effective gauge of a model's text comprehension capabilities.

A secondary rationale behind selecting these three text classification tasks pertains to their escalating difficulty level in terms of categories, as delineated in Table . The initial task, MARC-ja, revolves around binary sentiment classification. This is followed by Japanese Natural Language Inference (JNLI), a tri-category sentence pair classification task, and concludes with the Japanese Semantic Textual Similarity (JSTS), initially a 26-category task ranging from 0-5 at 0.2 intervals. For the sake of maintaining a linear progression in difficulty, we modified JSTS into a 6-category task with a 0-5 score range at 1 point intervals.

| **Model**         | **Parameters** | **Order** |
|:------------------|:---------------|:---------:|
| **T5-base**       | 220 Million    |  $10^8$   |
| **LLaMA-7B**      | 7 Billion      |  $10^9$   |
| **LLaMA-7B-LoRA** | 7 Billion      |  $10^9$   |
| **LLaMA-13B**     | 13 Billion     | $10^{10}$ |
| **GPT-3.5-Turbo** | —              |     —     |
| **GPT-4**         | —              |     —     |

Detail of Models.

## Model

Table presents several mainstream models selected based on their exponential size. To benchmark against PLM era models, we chose T5-base-japanese[^1], representative of the hundred-million-level. At the billion and ten-billion levels, we selected the open-source LLaMA model. Given that LLaMA is primarily trained on English corpora, its support for Japanese might be limited. For comparison, we included LLaMA-7B-LoRA[^2], a model

enhanced with training on Chinese and Japanese corpora using the LoRA adapter. This selection aids in assessing whether simply training on Chinese and Japanese corpora improves Japanese task performance. Our approach ensured test models with exponentially increasing parameters, attempting to corroborate that model performance also increases linearly.

## Prompt Template Design

In this section, we delve into the process of crafting distinct prompt templates to evaluate the performance of the language model concerning Japanese linguistic features. As demonstrated in Figure 2, we utilize the initial sentiment classification dataset as an example, with templates numbered from 1 to 5, each representing a different design strategy.

Template No. 1 was conceived using the concept of Instruction Learning. Here, explicit instructions are appended to the basic questions to stimulate the language model's comprehension. However, this varies from the original paper which presented multiple choices(e.g., "A","B","C","D" ) for classification, requiring the model to output the same for prediction. To better align with real-world application scenarios, we adopted a narrative style of questioning where the options are seamlessly integrated into the question sentences.

Template No. 2 constitutes the fundamental prompt template, albeit devoid of the instructive sentence. In contrast, Template No. 3 retains the basic prompt structure but omits the distinctive Japanese honorific word "

"; a deletion that does not alter the overall sentence meaning.

Template No. 4 presents a minor change, with the question mark "?" replaced by the period "

". This modification reflects the characteristic of Japanese interrogative sentences which can end with either a question mark or a period.

Lastly, Template No. 5 maintains the original sentence meaning but modifies the syntax of the sentence's latter half. Our objective is to determine if these nuanced differences exert any significant impact on the model's ultimate performance.

# Evaluation

```latex
\begin{table*}[ht]\centering

\begin{tabularx}{1\textwidth}{@{}l*{6}{>{\centering\arraybackslash}X}>{\centering\arraybackslash}X>{\centering\arraybackslash}X>{\centering\arraybackslash}X>{\centering\arraybackslash}X>{\centering\arraybackslash}X>{\centering\arraybackslash}X@{}}
\toprule[2pt]
 & & & MARC-ja & &\\
Prompt Template & 1 & 2 & 3 & 4 & 5 & SD \\
\midrule
T5-base & 3.00(34.74) & 49.80(12.06) & 44.40(6.66) & 47.60(9.86) & 43.90(6.16) & 17.50\\
LLaMA-7B & 54.30(16.82) & 80.80(9.68) & 80.10(8.98) & 81.10(9.98) & 59.30(11.82) & 11.80\\
LLaMA-7B-LoRA & 82.40(0.38) & 84.60(2.58) & 83.60(1.58) & 83.90(1.88) & 75.60(6.42) & 3.29\\
LLaMA-13B & 83.00(13.02) & 65.00(4.98) & 73.40(3.42) & 66.30(3.68) & 62.20(7.78) & 7.48\\
GPT-3.5-Turbo & 71.10(5.52) & 77.20(0.58) & 77.40(0.78) & 76.80(0.18) & 80.60(3.98) & 3.08\\
GPT-4 & \textbf{88.70}(0.26) & \textbf{88.70}(0.26) & \textbf{86.20}(2.24) & \textbf{88.50}(0.06) & \textbf{90.10}(1.66) & 1.26\\
\bottomrule[2pt]
\end{tabularx}
\vspace{1em}

\begin{tabularx}{1\textwidth}{@{}l*{6}{>{\centering\arraybackslash}X}>{\centering\arraybackslash}X>{\centering\arraybackslash}X>{\centering\arraybackslash}X>{\centering\arraybackslash}X>{\centering\arraybackslash}X>{\centering\arraybackslash}X@{}}
\toprule[2pt]
 & & & JNLI & &\\
Prompt Template & 1 & 2 & 3 & 4 & 5 & SD \\
\midrule
T5-base & 1.00(0.70) & 0.30(0.00) & 0.00(0.30) & 0.20(0.10) & 0.00(0.30) & 0.37\\
LLaMA-7B & 14.40(7.68) & 18.70(3.38) & 37.10(15.02) & 17.20(4.88) & 23.00(0.92) & 8.01\\
LLaMA-7B-LoRA & 15.10(8.98) & 15.70(8.38) & 18.30(5.78) & 17.10(6.98) & \textbf{54.20}(30.12) & 15.10\\
LLaMA-13B & 19.60(6.14) & 11.10(2.36) & 13.00(0.46) & 10.80(2.66) & 12.80(0.66) & 3.19\\
GPT-3.5-Turbo & 31.60(12.6) & \textbf{49.70}(5.5) & \textbf{43.10}(1.1) & \textbf{48.70}(4.5) & 47.90(3.70) & 6.70\\
GPT-4 & \textbf{40.39}(6.75) & 25.54(8.10) & 27.62(6.02) & 25.44(8.2) & 49.21(15.57) & 9.56\\
\bottomrule[2pt]
\end{tabularx}
\vspace{1em}

\begin{tabularx}{1\textwidth}{@{}l*{6}{>{\centering\arraybackslash}X}>{\centering\arraybackslash}X>{\centering\arraybackslash}X>{\centering\arraybackslash}X>{\centering\arraybackslash}X>{\centering\arraybackslash}X>{\centering\arraybackslash}X@{}}
\toprule[2pt]
 & & & JSTS & &\\
Prompt Template & 1 & 2 & 3 & 4 & 5 & SD \\
\midrule
T5-base & 0.00(0.88) & 1.60(0.72) & 1.70(0.82) & 0.40(0.48) & 0.70(0.18) & 0.67\\
LLaMA-7B & 11.30(0.86) & 11.90(0.26) & 11.80(0.36) & 12.30(0.14) & 13.50(1.34) & 0.74\\
LLaMA-7B-LoRA & 29.30(10.62) & 14.30(4.38) & 10.10(8.58) & 15.80(2.88) & 23.90(5.22)  & 6.94\\
LLaMA-13B & 9.70(1.82) & 12.00(0.48) & 11.90(0.38) & 11.90(0.38) & 12.10(0.58) & 0.91\\
GPT-3.5-Turbo & 34.50(3.18) & 39.90(2.22) & 38.30(0.62) & 39.70(2.02) & 36.00(1.68) &
 2.11\\
GPT-4 & \textbf{49.20}(2.21) & \textbf{48.37}(1.38) & \textbf{50.70}(3.71) & \textbf{49.55}(2.56) & \textbf{37.11}(9) & 4.99\\
\bottomrule[2pt]
\end{tabularx}

\caption{\label{table5}
The accuracy of 0-shot experiment with different prompt templates for MARC-ja, JNLI, and JSTS. The value in "()" is the absolute deviation. SD is denote Standard Deviation.
}
\end{table*}
```

We employed a text classification methodology utilizing a question and answer format. Consequently, the generated output from the model could potentially be a complete sentence, such as "It's positive," which incorporates the designated label word, in this case, "positive." To extract this classified label word from the generated response text, we devised a straightforward algorithm. For the assessment of model performance, we selected accuracy as the metric to facilitate evaluation. The pseudo-code delineating the process of extracting label words from the generated text can be located in Appendix .

Moreover, to highlight the variability of accuracy across different models for varying prompt templates, we conducted a statistical analysis. Assuming the model's accuracy on prompt templates No.1-5 as $(x_1, x_2, x_3, x_4, x_5)$, we calculated their mean value, denoted as $M$. Subsequently, we determined the absolute deviation score by calculating the absolute value of the difference between each $x$ value and $M$. Through the computation of the absolute deviation score, one can effectively illustrate the extent to which each accuracy deviates from the average.

$$M = \frac{x_1 + x_2 + x_3 + x_4 + x_5}{5}$$

$$\begin{aligned}
\text{Absolute Deviation Score} &= |x_i - M| \nonumber \\
&\text{for } i = 1,2,3,4,5
\end{aligned}$$

In addition to this, we computed the standard deviation of a set of accuracies for each model, thereby providing a statistical depiction of the accuracy fluctuations across different prompt templates for each model.

# Results

In this section, we present a segmented analysis of the experimental results, divided according to the specific datasets used. Each segment will be examined individually. Notably, an in-depth exploration of the results concerning the T5-base will be provided in Section as a representative example.

Table 1 summarizes the outcomes of all conducted experiments. A review of the overall results reveals a clear correlation between the high and low accuracy rates across data sets and the performance capabilities of each model. Nonetheless, a number of anomalous findings emerged. These unexpected results offer valuable insights, potentially highlighting areas of concern within the current LLMs.

## MARC-ja

In the basic sentiment binary classification task using the MARC-ja dataset, the GPT-4 model demonstrated the lowest standard deviation at 1.26. This implies that GPT-4's performance across different prompt templates was relatively consistent when compared to other models. However, there were noticeable deviations in the performance of GPT-4, with an absolute deviation score of 2.24 for Prompt Template 3 and 1.66 for Prompt Template 5. It was observed that the omission of an honorific from Prompt Template 3 resulted in a decrease in accuracy to 86.20. Conversely, a modification in the sentence structure of Prompt Template 5 led to an increased accuracy of 90.10. This data suggests that the stability of GPT-4 when processing various prompt templates remains an area for improvement.

Secondly, the LLaMA-7B-LoRA model, trained using the Sino-Japanese corpus, demonstrated a significant reduction in standard deviation to 3.29 approximately half that of LLaMA-13B. Furthermore, its accuracy outstripped that of LLaMA-7B by a remarkable 28.1 points on the instruct Prompt Template 1. This underscores the model's capability to enhance the comprehension of minor languages like Japanese, given expanded training on Chinese and Japanese corpus. Yet, after the change in Prompt Template 5, LLaMA-7B-LoRA experienced notable volatility, with accuracy tumbling to 75.60. Despite this, the LLaMA-7B-LoRA model's overall performance surpassed that of GPT-3.5-Turbo on the MARC dataset.

The subsequent findings for LLaMA-13B illustrate a compelling phenomenon. It appears that nearly doubling the size of the model precipitates a substantial enhancement in the accuracy of instruct Prompt Template 1, which escalates dramatically to 83. This finding suggests a proportional correlation between the model's size and its ability to comprehend the problem more proficiently. The improvement in performance is indicative of the critical role instruct learning plays, thus reaffirming its influence in the LLMs learning process.

## JNLI

Upon increasing the task difficulty to triple classification in the JNLI dataset, a significant performance drop is observed across all models. The LLaMA-13B and T5-base models exhibit small standard deviations, which could be attributed to their low accuracy. The performance degradation for the remaining four models is even more pronounced. Notably, the standard deviations for GPT-3.5-Turbo and GPT-4 models escalated to 6.7 and 9.56, respectively. Moreover, the GPT-3.5-Turbo model managed to surpass the GPT-4 model by approximately twofold in the accuracies of Prompt Templates 2-4. Nevertheless, despite these fluctuations, both the GPT-3.5-Turbo and GPT-4 models consistently demonstrated superior performance when compared to the LLaMA model.

## JSTS

In summary, there is a noticeable reduction in the standard deviation when dealing with the complex six-category JSTS dataset. The ranking of model accuracy, both at its highest and lowest, appears to be aligned with the inherent strengths and weaknesses of the respective models. It is notable that the GPT-3.5-Turbo model demonstrated a significant decline in performance, with the accuracy dropping to 34.5 when utilizing construct Prompt Template 1. Similarly, the GPT-4 model experienced a substantial decrease, with accuracy falling to 37.11 upon employing the modified Prompt Template 2. Nevertheless, the complexity of the JSTS dataset, particularly with the sentence type alterations in the modified Prompt Template 5, exceeded those in the previous four prompts template. This increased complexity led to the observed decline in accuracy.

# Analysis

In this section, we present a comprehensive analysis and interpretation of the experimental results, aiming to enhance their intuitiveness. The analysis of these results will be conducted in two parts. The first subsection involves analysis through the use of line graphs and box-plots to demonstrate the data's distribution and trends. In the second subsection , we integrate the actual textual content of the generated results into our analysis, providing a more contextual understanding of the findings.

## Analysis with Plot

Figure presents three line graphs, each corresponding to a different difficulty dataset. The horizontal axis denotes various models, while the vertical axis indicates accuracy. Ideally, there should be a broadly increasing trend aligned with model performance strength. Yet, as demonstrated in Figure , Prompt Template 1 exhibits robust performance with both the LLaMA series and GPT-4 models on the MARC-ja dataset, despite a decrease in accuracy with the GPT-3.5-Turbo model. Upon examining the other four prompt templates of GPT-3.5-Turbo, it becomes evident that adding an instructive sentence to Prompt Template 1 results in a performance degradation.

Figure demonstrates a significant increase in the model's sensitivity towards the prompt template, particularly as the classification task complexity escalates. Notably, the model's robustness seems to exhibit considerable variability. Nevertheless, it is essential to highlight that of all the prompt templates, Prompt Template 1 remains the most stable. The accuracy fold displays a steady upward trend, which underscores the efficacy of instruction learning for LLMs. These models exhibit a superior capacity for understanding long text problems in comparison to the T5-base model. However, the scenario is different for GPT-3.5-Turbo and GPT-4. There are prominent fluctuations across multiple prompt templates, suggesting that even minor differences can lead to significant variations in outcomes, especially for slightly more complex tasks. This scenario is particularly prevalent when ChatGPT products are in use; user's habitual usage of a specific prompt template may yield effective results on one model, but switching to a different model may result in a subpar performance. This issue warrants immediate attention, as it presents a pressing challenge for the model's overall utility and effectiveness.

As the complexity of the task escalates incrementally, the GPT series model consistently outperforms the LLaMA series models, as clearly depicted in Figure . Furthermore, the GPT series model demonstrates superior stability when processing diverse prompt templates, with negligible fluctuation observed.

To more effectively illustrate the degree of fluctuation exhibited by various models when processing different prompt templates, we utilized box-plots. These diagrams offer a visual representation of model fluctuation for each prompt template, with the area of the box indicating the extent of the variation. As demonstrated in Figure , certain data points are observed outside the boundaries of the box-plot. This occurrence can be attributed to the nature of the box-plot representation, which intentionally excludes certain outliers, thereby providing a more accurate depiction of the underlying data's volatility. The box-plot features additional extended lines, which represent the maximum and minimum values once outliers have been excluded.

As depicted in Figure , the LLaMA-7B-LoRA model achieves performance parity with the GPT series, even exceeding the accuracy of GPT-3.5-Turbo. This demonstrates the efficacy of LLaMA-7B-LoRA, particularly after training with Chinese-Japanese language enhancement, in significantly improving performance for the less commonly used Japanese language. This improvement is noteworthy, as it is achieved without the necessity for a vast quantity of parameters, unlike ChatGPT.

In contrast, Figure reveals that despite the significant performance enhancement of the GPT series when compared to other models, there is a marked increase in the area of the box-plot. This suggests potential robustness issues when these models handle different prompt templates.

Lastly, Figure shows that with the exception of the LLaMA-7B-LoRA model, which displays significant stability fluctuations, the box-plot area of all other models remains relatively small, indicating lesser degrees of fluctuation.

## Analysis with Generated Text

Figure illustrates the specific results generated from Prompt Templates 1 and 2 applied to the MARC-ja dataset. A notable observation is that the T5 model fails to comprehend the overarching meaning of the question, hence being unable to produce a relevant answer. Instead, it merely replicates the sentence following "Answer:" from the input text as the output. This outcome is likely attributed to the model's limited size, which results in the generation of tokens based on maximum probability, rather than contextual understanding.

In contrast, LLMs, such as LLaMA-7B and LLaMA-13B, successfully interpret the questions and respond appropriately. Despite the lack of native support for Japanese within these models, they manage to generate meaningful responses. However, they occasionally produce unique tokens, which is likely a byproduct of the linguistic limitation. Furthermore, the incorporation of the Japanese corpus in the LLaMA-7B-LoRA model training significantly enhances its effectiveness in generating Japanese text.

Finally, it is worth highlighting the difference in language style between GPT-3.5-Turbo and GPT-4. GPT-3.5-Turbo's output bears a closer resemblance to conversational language, while GPT-4 adeptly generates responses in a format aligned with the specifications given in the question.

## Limitations

This work was tested only for Japanese. It was not tested for other minor languages. Furthermore, due to limitations in time and resources, this experiment could not be conducted utilizing the full extent of the Japanese text classification dataset. Secondly, the experiments were conducted only for the basic text classification task. And no other natural language processing tasks were tested. In terms of model selection, we opted for the most recent and representative LLMs. However, we acknowledge that this choice may not yield extensive coverage. Despite these limitations, we posit that our approach helped control extraneous variables, thus presenting a more realistic, controlled experimentation environment that closely reflects real-world conditions.

# Conclusion

This study endeavors to evaluate the sensitivity and resilience of existing Large Language Models (LLMs) in response to varying prompt templates, utilizing a basic Japanese benchmark dataset as a test case. The experimental findings indicate four key insights: 1) Current LLMs exhibit a degree of instability, with even slight variations in the prompt template potentially leading to substantial fluctuations in model output; 2) Uniform applicability of a single prompt template across all LLMs is not guaranteed; 3) Instruction-based learning proves to be more effective for some LLMs(e.g., LLaMA-13B) when compared to the use of a common prompt template; 4) LLMs require additional training to bolster their robustness when dealing with the Japanese language.

These findings pave the way for a significant research direction for LLM developers and researchers, namely, **enhancing the resilience of LLMs to diverse prompt templates when dealing with less commonly studied languages**. We assert that addressing this issue is vital for the future progression of LLMs and represents a challenge that necessitates resolution.

# Prompt Template in Native Speaker

As shown in Figure , it is a more native Japanese question sentence. But in this experiment, we did not use this sentence as prompt template. First of all, Japanese and Chinese belong to the same Kanji circle. If we could replace the positive and negative words with Chinese words that also exist in the Chinese language. The model might be able to gain cross-linguistic knowledge from the richer Chinese corpus to enhance the understanding on Japanese. So we replaced several kanji words with kanji words common to both Chinese and Japanese in the native language representation here.

# Prompt Template of JNLI and JSTS

As shown in Figure , the prompt template we designed for the JNLI dataset. The design idea of the five different prompt templates for the three datasets is the same. A category of words is added to the previous MARC-ja dataset.

In contrast, Figure demonstrates a slight deviation from this pattern due to the inherent structure of the JSTS dataset, which operates on a scoring system. To augment the model's comprehension of pure scores, we incorporated a six-segment evaluation component specific to the Japanese language into the prompt template. Consequently, this adaptation results in a template that exhibits slight differences from the ones designed for the prior two datasets.

# Generated label extraction algorithm

As shown in algorithm 1, we design a simple algorithm to extract generated labels and calculate accuracy. First, the total number of samples extracted from each data set is 1000. Then identify and extract the label word from the answer. accuracy is then calculated based on the extracted lable and the actual label.

[^1]: <https://huggingface.co/sonoisa/t5-base-japanese>

[^2]: <https://huggingface.co/KBlueLeaf/guanaco-7b-leh-v2>


abstract: We present and compare two high-pressure, high-frequency electric-discharge sources for the generation of supersonic beams of fluorine radicals. The sources are based on dielectric-barrier-discharge (DBD) and plate-discharge units attached to a pulsed solenoid valve. The corrosion-resistant discharge sources were operated with fluorine gas seeded in helium up to backing pressures as high as 30 bar. We employed a (3+1) resonance-enhanced multiphoton ionization combined with velocity-map imaging for the optimization, characterization and comparison of the fluorine beams. Additionally, universal femtosecond-laser-ionization detection was used for the characterization of the discharge sources at experimental repetition rates up to 200 Hz. Our results show that the plate discharge is more efficient in F$_{2}$ dissociation than the DBD by a factor of 8-9, whereas the DBD produces internally colder fluorine radicals.
author: Patrik Straňák; Ludger Ploenes; Simon Hofsäss; Katrin Dulitz; Frank Stienkemeier; Stefan Willitsch
date: 2024-10-03
title: Development and characterization of high-frequency sources for supersonic beams of fluorine radicals

# Introduction

As the element with the highest electronegativity, fluorine is reactive with a wide range of chemical compounds. Due to its high reactivity, atomic fluorine is often used in reaction-dynamics studies of fast radical reactions. For instance, recent studies show its significance in revealing dynamical resonances and resonance-induced quantum tunneling in the reaction F+H$_{2}$. The mechanisms and dynamics of reactions with fluorine radicals in the gas phase are often studied with crossed-molecular-beam (CMB) techniques which rely on the generation of cold, supersonic beams with high densities.

In early crossed-beam experiments, beams of fluorine radicals were produced by thermal dissociation or microwave discharge of F$_2$ or other fluorine-containing precursors. In order to narrow the large spread of the beam velocity inherent in these approaches, velocity selectors were used which, however, also decreased the number densities of radicals and, therefore, reduced the advantage of an efficient production of F radicals afforded by these techniques. Nesbitt and co-workers generated fluorine radicals by implementing a plate-discharge source at the orifice of a pulsed-gas nozzle. Yang and co-workers further improved the discharge efficiency by introducing a double-stage pulsed discharge source, the stability of which was enhanced by a pre-ionization stage in the discharge. Regular maintenance of the valves was a necessity due to constant flow of highly corrosive gas of molecular fluorine through the moving mechanism of the plunger. Liu and co-workers addressed this problem by separating the actuation mechanism of the plunger from the gas flow by modifying a pulsed piezoelectric-disk valve resulting in increased operation times. Typically, the repetition rates of previous pulsed fluorine-radical sources were rather low, reaching up to 20 Hz. An increase of the repetition rate was recently achieved by the development of a discharge source based on a piezoelectric stack actuator by Suits and co-workers that allowed the use of much higher actuation forces than conventional piezo disks. The backing pressures used in these pulsed fluorine discharge sources were relatively low. Benefits of higher backing pressures, such as increased beam densities, narrower velocity spreads and internally colder beams, were achieved by combining Even-Lavie-type gas valves with a DBD or a conventional plate discharge resulting in atomic beams of a wide range of different radicals.

Here, we present a corrosion-resistant atomic fluorine source based on a pulsed solenoid valve which can be operated at high backing pressures and high operation frequencies. The use of corrosion-resistant materials, such as stainless steel, macor and teflon, enables stable operation for months with little to no maintenance. We compare and characterize in detail two different dissociation techniques, a conventional plate discharge and a dielectric-barrier discharge, both of which are often used in crossed-beam experiments for the production of radicals, metastable species or ions.

The high ionization energy of atomic fluorine of 17.4 eV poses challenges for the characterization of fluorine radical sources. In the past, the direct detection of F involved either electron paramagnetic resonance spectroscopy or the generation of vacuum-ultraviolet (VUV) radiation for single- or multi-photon ionization detection schemes, both of which were associated with challenging requirements for the experimental setup. Incomplete dissociation of molecular fluorine in the discharge requires the ability to differentiate between fluorine originating from the discharge process and from spurious photodissociation by the probe laser for an accurate characterization of the radical beam. Various techniques have previously been used for the indirect detection of fluorine atoms, e.g., by titration with Cl$_{2}$, by elastic scattering from an inert gas followed by electron bombardment or by probing the products of reactive collisions. These rather slow detection methods are, however, not ideal in the context of optimization procedures requiring a fast response from the experiment. Here, we present the adaption of a (3+1) resonance-enhanced multiphoton ionization (REMPI) scheme employed by Roth et al. and universal femtosecond-laser ionization in combination with time-of-flight mass spectrometry and velocity-map imaging as a tool for the optimization and characterization of the present fluorine-radical sources.

# Experimental setup

The discharge sources were characterized and optimized using a CMB apparatus recently set up in our laboratory. A schematic of the part of the apparatus relevant for the current experiments is depicted in Fig. . It consisted of two differentially pumped vacuum chambers separated by a conical skimmer. The source and the detection chambers were pumped by two and one turbo-molecular pumps (Oerlikon Leybold MAG W 2200), respectively, with pumping speeds of 2200 ls$^{-1}$ each. A first beam skimmer (orifice diameter 2 mm) was placed 11 cm downstream from the discharge source nozzle. A second skimmer with an orifice diameter of 1.5 mm was placed in front of the detection region to further confine the beam. Electrostatic deflection plates were placed in between the nozzle and the first skimmer in order to clean the beam from spurious ions produced in the discharge. The total distance from the nozzle to the intersection region with the probe laser amounted to 46 cm.

The discharge units were attached to a solenoid-based pulsed valve designed in the Canadian Center for Research on Ultra-Cold Systems (CRUCS). The valve was mounted on an $xyz$-translational stage inside the source chamber. The duration for opening the valve during a gas pulse was set to be nominally $32~\mu$s identical for both discharge sources to enable a direct comparison. Typical operating pressures in the source and detection chamber at 20 Hz repetition rate were in the low 10$^{-6}$ and 10$^{-8}$ mbar regime, respectively. An increase of the repetition rate of the experiment to 200 Hz increased the pressures in both chambers by a factor of 10. A molecular beam (MB) of fluorine was formed by a supersonic expansion of a commercially available mixture of 5% F$_{2}$ in He (Carbagas AG). The F radicals were created by dissociation of the F$_2$ molecules using the electric-discharge sources.

The design of the discharge units was based on sources originally developed for the generation of dense beams of OH radicals. Construction details of the sources are depicted in Fig. . For the DBD source \[Fig.  (a)\], the electric discharge takes place at the surface of a dielectric material (Macor^^) in the form of microdischarge channels between the inner electrode (a magnetic disc) and the valve body. The DBD is generated by applying an AC voltage to the electrode, with a peak-to-peak amplitude of 4.5 kV and a frequency of 780 kHz. The amplitude was limited by the commercial driver used (Lamid Ltd.). The optimal delay of the discharge pulse relative to the valve opening trigger was found to be $30~\mu$s.

For the plate-discharge source \[Fig. (b)\], the electric discharge is created between two metal plates isolated by electric insulators (Macor^^) by applying a voltage of -1kV to the electrode intersecting the nozzle orifice. A home-built high-voltage driver was used for igniting the discharge. Because the distance between the electrodes and the valve orifice is increased compared to the DBD source, the optimum delay between the gas and discharge pulses had to be raised to $55~\mu$s compared to the DBD source.

The addition of an electron-emitting filament in front of the nozzle, as reported in Ref. , increased the stability of the DBD source. This was not required for the plate discharge. Since the filament was also introducing additional heat into the system, it was nonetheless kept operational also for the plate discharge in order to maintain the setup at the same temperature and thus provide the same conditions for a direct comparison between the discharge sources.

Fluorine radicals were state-selectively ionized using a (3+1) REMPI scheme following Ref. . The output of a pulsed dye laser (Liop-Tec LiopStar E) pumped by a Nd:YAG laser (Innolas Spitlight 1000) at a repetition rate of 20 Hz was frequency doubled by a $\beta$-barium borate (BBO) crystal to produce radiation at a wavelength of 234 nm. The laser beam entered and exited the chamber via two Brewster windows in order to minimize stray light. The light was focused into the interaction region by a lens with a focal length of 70 mm placed inside the vacuum chamber. The polarization of the light was chosen parallel to the imaging plane. Measurements at higher repetition rates up to 200 Hz were performed using universal 775 nm ionization with a femtosecond laser (Clark MXR) with a pulse duration of 150 fs.

The detection of the ionization products was performed by time-of-flight mass spectrometry (TOF-MS) and mass-specific velocity-map imaging (VMI). The ions were accelerated towards a double-stack microchannel-plates (MCP) detector connected to a phosphor (P46) screen (Photek, Ltd). In VMI mode, mass selection was achieved by time-gating the MCPs. In TOF-MS mode, the electrodes were operated in Wiley-McLaren configuration by applying $U_1$ = 3000 V and $U_2$ = 2000 V to the extraction plates. To achieve optimal velocity focusing in VMI mode, $U_2$ was increased to 2235 V.

Both discharge sources were operated for two hours before taking any measurements to reach stable experimental conditions. Additionally, to ensure a fair comparison of both discharge mechanisms, the density of the neat molecular beam prior to dissociation was verified to be constant in all experiments by monitoring the number of F ions produced by the (3+1) REMPI ionization of F$_2$.

# Results and discussion

## (3+1) REMPI spectrum of F atoms

As a preparation for the characterization and optimization of the discharge sources, (3+1) REMPI spectra of fluorine atoms originating from the photodissociation of F$_{2}$ in the absence of the discharge were recorded. At 234 nm, photodissociation of F$_{2}$ $$\label{eq1}
\text{F$_{2}$} + hv \rightarrow \text{2F($^{2}$P$_{J}$)},$$ occurs via the repulsive state A$^{1}\Pi_\text{u}$. Fluorine atoms generated in different spin-orbit components $^{2}$P$_{J}$, where $J$ denotes the quantum number of the total atomic angular momentum excluding nuclear spin, were further ionized by (3+1) REMPI using the same wavelength and the resulting ion signal was detected using TOF-MS.

An example of a REMPI spectrum is shown in Fig. . The one-photon wavenumber was scanned from 42590 to 42790 cm$^{-1}$ at a pulse energy of 0.3 mJ in the interaction region. The spectrum exhibits transitions from the ground (2p$^{5}$) $^{2}$P$_{3/2}$ and spin-orbit excited (2p$^{5}$) $^{2}$P$_{1/2}$ states to the (2p$^{4}$`<!-- -->`{=html}3d) $^{2}$D$_{j}$ and $^{2}$P$_{k}$ levels by the non-resonant absorption of three photons. A slight systematic shift of $\approx 3$ cm$^{-1}$ of the transition wavenumbers in Fig.  in comparison to the literature values is attributed to differences in the wavelength calibration (here, a HighFinesse WS-6 wavemeter with an internal Ne calibration lamp was used) and Doppler shifts due to the oblique positioning of the laser with respect to the molecular beam.

## Comparison of discharge sources using velocity-map imaging

The discharge sources were optimized by measuring the yield of fluorine radicals on the $^{2}$P$_{3/2}\rightarrow^{2}$D$_{5/2}$ (3+1) REMPI transition (Fig. ) using VMI. The velocity-resolved images allowed us to distinguish F radicals produced in the discharge source from atoms generated by the photodissociation of F$_{2}$ by the REMPI laser. F atoms arising from the photodissociation process are generated with high recoil velocities and thus appear as rings in the VMI images. By contrast, F radicals produced by the discharge possess near zero recoil velocity after laser ionization and thus appear as a spot in the center of the VMI images. Exemplary results of these measurements are shown in Figs. (a)-(c) which compare VMI images obtained from a neat F$_2$/He beam with ones obtained using the DBD and plate discharges.

Integration of the VMI images over the angular coordinate resulted in the radial product distributions shown in Figs. (d)-(f). In contrast to the neat beam \[Fig. (a) and (d)\], for both the DBD- and plate-discharged beam a clear signal was observed in the center of the images which is attributed to the fluorine radicals generated by the discharges. This is accompanied by a decrease in fluorine signal originating from the photodissociation of F$_2$.

Both discharge sources were measured under their respective optimized conditions established by maximizing the fluorine signal accumulated in the centre of the images. For the DBD source, the maximum F yield was reached at a valve backing pressure of 10 bar, whereas for the plate-discharge source, the highest yields were achieved at a backing pressure of 30 bar. A further increase of the backing pressure was not attempted due to limitations of the corrosion-resistant pressure regulator used. The discharge efficiencies of the DBD and the plate discharges were compared by integrating the fluorine signal in the center of the images, while taking into account a small background contribution. Based on this comparison, the radical yield of the plate discharge was estimated to be 8-9 times higher than that of the DBD. Using the same backing pressure of 10 bar for both sources, the plate discharge still produces 2-3 times higher fluorine atom fluxes. Alternatively, a comparison of the ratios of fluorine radicals produced by the discharge and by photodissociation, as shown in the radial product distributions (d), (e) and (f) of Fig. , gives a factor of 7.2 in radical yield between the plate-discharge and the DBD sources which is similar to the result obtained from comparing the integrated signals of the central spots. These results indicate a markedly higher efficiency of the plate discharge compared to the DBD for producing F atoms.

## Relative state populations of F radicals generated by the two methods

By comparing the intensity of different transitions in the (3+1) REMPI spectra of the F atoms generated in the discharge sources, the relative populations of the ground $^{2}$P$_{3/2}$ and spin-orbit excited $^{2}$P$_{1/2}$ states were determined by integrating the relevant signal peaks for the DBD and plate discharge (Fig. ). The plate-discharge source produces more radicals in the spin-orbit excited $^{2}$P$_{1/2}$ state, indicating an internally hotter radical beam compared to the DBD source. Similar results were previously obtained for the generation of OH radicals with both discharge techniques.

## Operation of the sources at high experimental repetition rates

In order to test the performance of the plate-discharge source at various experimental repetition rates up to 200 Hz, VMI images obtained by femtosecond-laser ionization were recorded. Figure shows an example of VMI images of the (a) neat and (b) discharged beam at a repetition rate of 100 Hz. With fs-laser ionization, the VMI images appear highly structured containing a number of features. These are associated with fs-laser multiphoton dissociation of molecular fluorine producing F atoms in a variety of internal states. In contrast to (3+1) REMPI ionization, the multiphoton excitation and ionization by the fs laser partly also produces slow F$^+$ ions which appear in the center of the image and mask the contribution of F atoms generated by the discharge, as can be seen in Fig. (a).

Radial product distribution functions obtained from the VMI images taken at 20, 100 and 200 Hz \[Figs. (c)-(e)\] show no significant difference in the performance of the discharge source as a function of the repetition rate. Achieving even higher repetition rates was only prevented by the increased heating of the valve caused by the driving current of the valve mechanism and the plunger friction. Additional cooling of the valve should enable to overcome this limitation and reach even higher repetition rates. A similar constant performance as a function of the repetition rate was observed for the DBD source (not shown here). Analysis of the ion counts in these images indicates that the plate-discharge source produces about a factor of 7 more F radicals than the DBD source at each repetition rate studied, in good agreement with the results obtained by (3+1) REMPI.

## Velocity profiles of the radical beams

To obtain velocity profiles of the fluorine radical pulses, VMI images were measured at a repetition rate of 100 Hz for different time delays of the gas beam with respect to the trigger of the fs laser. Velocities of the beam at each time delay were extracted from these measurements. Normalized velocity profiles of the radical beam profiles after subtracting the contribution from photodissociated fluorine atoms are shown in Fig.  for both discharge sources. The profiles are very similar. The speed ratio estimated from the velocity at maximum intensity with $\Delta v$, the full width at half maximum, was determined to be $v/\Delta v\approx 14$ and 18 for the DBD and plate-discharge source, respectively. A slightly faster velocity was observed for the beam produced by the plate discharge which is attributed to the more violent character of this type of discharge.

# Summary and conclusions

We have presented two corrosion-resistant discharge sources, a high-voltage plate discharge and a dielectric-barrier discharge, for the generation of beams of atomic fluorine radicals operational at repetition rates up to 200 Hz and high backing pressures. Their performance was analyzed with the help of velocity-map ion imaging of the generated F atoms and compared in terms of their efficiencies and beam properties. The combination of the discharge sources with a high-pressure, short-pulse solenoid valve produced high-density atomic beams of comparable narrow longitudinal velocity spreads. It was found that the plate discharge produces denser fluorine radical beams at the cost of increased internal temperature, as has also been observed in the production of OH radicals. Both discharge sources exhibit an excellent resistivity to corrosion thus offering long-term stable operation without the requirement of frequent maintenance. The sources were intensively used for the production of fluorine radicals over several months with only requiring the exchange of teflon gaskets every few weeks. Both sources showed no dependence of their discharge efficiency on the experimental repetition rate up to 200 Hz making them suitable for experiments running at high frequencies.

We note that the present plate-discharge source was also recently successfully applied in a study of the chemi-ionization reaction of metastable neon atoms with carbonyl sulfide molecules.

# Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.


abstract: The issue of fairness in recommendation is becoming increasingly essential as Recommender Systems (RS) touch and influence more and more people in their daily lives. In fairness-aware recommendation, most of the existing algorithmic or in-processing approaches mainly aim at solving a constrained optimization problem by imposing a constraint on the level of fairness while optimizing the main recommendation objective, e.g., click through rate (CTR). While this alleviates the impact of unfair recommendations, the expected return of an approach may significantly compromise the recommendation accuracy due to the inherent trade-off between fairness and utility. This motivates us to deal with these conflicting objectives and explore the optimal trade-off between them in recommendation. One conspicuous approach is to seek a *Pareto efficient/optimal* solution to guarantee optimal compromises between utility and fairness. Moreover, considering the needs of real-world e-commerce platforms, it would be more desirable if we can generalize the whole *Pareto Frontier*, so that the decision-makers can specify any preference of one objective over another based on their current business needs. Therefore, in this work, we propose a fairness-aware recommendation framework using *multi-objective reinforcement learning* (MORL), called MoFIR (pronounced "**more fair**"), which is able to learn a single parametric representation for optimal recommendation policies over the space of all possible preferences. Specially, we modify traditional DDPG by introducing *conditioned network* (CN) into it, which conditions the networks directly on these preferences and outputs Q-value-vectors. Experiments on several real-world recommendation datasets verify the superiority of our framework on both fairness metrics and recommendation measures when compared with all other baselines. We also extract the approximate Pareto Frontier on real-world datasets generated by MoFIR and compare to state-of-the-art fairness methods.
bibliography: sample-base.bib
title: Toward Pareto Efficient Fairness-Utility Trade-off in Recommendation through Reinforcement Learning

# Introduction

Personalized recommender systems (RS), which are extensively employed in e-commerce platforms, have been acknowledged for their capacity to deliver high-quality services that bridge the gap between products and customers. Despite these huge advantages, several recent studies also raised concerns that RS may be vulnerable to algorithmic bias in several aspects, which may result in detrimental consequences for underrepresented or disadvantaged groups . For example, the "Matthew Effect" becomes increasingly evident in RS, which creates a huge disparity in the exposure of the producers/products in real-world recommendation systems . Fortunately, these concerns about algorithmic fairness have resulted in a resurgence of interest to develop fairness-aware recommendation models to ensure such models do not become a source of unfair discrimination in recommendation.

In the area of fairness-aware recommendation, the methods can be roughly divided into three categories: pre-processing, in-processing and post-processing algorithms . Pre-processing methods usually aim to remove bias in data, e.g., sampling from data to cover items of all groups or balancing data to increase coverage of minority groups; however, we often do not have access to the data collection process, but are given the dataset. In-processing methods aim at encoding fairness as part of the objective function, while post-processing methods tend to modify the presentations of the results. Even though all of them successfully alleviate the impact of unfair recommendations, the expected return of an approach may significantly compromise the recommendation accuracy due to the inherent trade-off between fairness and utility, which has been demonstrated by several recent work both empirically and theoretically .

In light of the above, one fundamental research question is **Q1**: *can we learn a recommendation model that allows for higher fairness without significantly compromising recommendation accuracy?* And a more challenging one is **Q2**: *can we learn a single recommendation model that is able to produce optimal recommendation policies under different levels of fairness-utility trade-off so that it would be more desirable for decision-makers of e-commerce platforms to specify any preference of one objective over another based on their current business needs.* To deal with **Q1**, one conspicuous approach is to seek a Pareto optimal solution to guarantee optimal compromises between utility and fairness, where a Pareto optimal solution means no single objective can be further improved without hurting the others. To find solutions with different levels of trade-off between utility and fairness we need to generalize their Pareto frontier in the objective space, where Pareto frontier denotes a set, whose elements are all Pareto optimal. Unfortunately, state-of-the-art approaches of fairness-aware recommendation are limited in dealing with the fairness-utility trade-off.

Therefore, in this work, we aim to address the above problems and propose a fairness-aware recommendation framework using multi-objective reinforcement learning (MORL) with linear preferences, called MoFIR, which aims to learn a single parametric representation for optimal recommendation policies over the space of all possible preferences. Technically, we first formulate the fairness-aware recommendation task as a Multi-Objective Markov Decision Process (MOMDP), with one recommendation objective, e.g., CTR, and one fairness objective, e.g., item exposure fairness (our method is able to deal with more recommendation objectives as well as more fairness objectives). Second, we modify classic DDPG algorithm by introducing conditioned networks into it. Specially, we condition the policy network and the value network directly on the preferences by augmenting them to the feature space. Finally, we utilize the vectorized Q-value functions together with modified loss function to update the parameters. The contributions of this work can be summarized as follows:

-   We study the problem of Pareto optimal/efficient fairness-utility trade-off in recommendation and extensively explore their Pareto frontier to better satisfy real-world needs;

-   We formulate the problem into a MOMDP and solve it through a MORL framework, MoFIR, which is optimized over the entire space of preferences in a domain, and allows the trained model to produce the optimal policy for any user-specified preferences;

-   Unlike prior methods for fairness-aware recommendation, the proposed framework does not employ any relaxation for objectives in the optimization problem, hence it could achieve state-of-the-art results;

-   Experiments on several real-world recommendation datasets verify the superiority of our framework on both fairness measures and recommendation performance when compared with all other baselines.

# Related Work

## Fairness in Recommendation

There have been growing concerns on fairness in recommendation as recommender systems touch and influence more and more people in their daily lives. Several recent works have found various types of bias in recommendations, such as gender and race , item popularity , user feedback  and opinion polarity . There are two primary paradigms adopted in recent studies on algorithmic discrimination: individual fairness and group fairness. Individual fairness requires that each similar individual should be treated similarly, while group fairness requires that the protected groups should be treated similarly to the advantaged group or the populations as a whole. Our work focuses on the item popularity fairness from a group level, yet it can used to solve multiple types of fairness simultaneously by properly defining and adding them as additional objectives.

The relevant methods related to fairness in ranking and recommendation can be roughly divided into three categories: pre-processing, in-processing and post-processing algorithms . First of all, pre-processing methods usually aim to minimize bias in data as bias may arise from the source data. This includes fairness-aware sampling methodologies in the data collection process to cover items of all groups, or balancing methodologies to increase coverage of minority groups, or repairing methodologies to ensure label correctness, remove disparate impact . However, most of the time, we do not have access to the data collection process, but are given the dataset. Secondly, in-processing methods aim at encoding fairness as part of the objective function, typically as a regularizer . Finally, post-processing methods tend to modify the presentations of the results, e.g., re-ranking through linear programming or multi-armed bandit . However, there is no free lunch, imposing fairness constraints to the main learning task, introduces a trade-off between these objectives, which have been asserted in several studies. For example, showed that because of noise on the underrepresented groups the trade-off between accuracy and equality of opportunity exists.

Unfortunately, there is very few work of fairness-aware recommendation can be found to study the fairness-utility trade-off. The closest one to our work is , which mainly focused on the trade-off between two-sided fairness in e-commerce recommendation. used a traditional multiple gradient descent algorithm to solve multi-objective optimization problem, meaning they need to train one network per point on the Pareto frontier, while our MoFIR generates the full Pareto frontier of solutions in a single optimization run. Besides, the authors relaxed all their objectives to get their differentiable approximations, which, to some extent, hurt its performance, as is shown in the experiment part, Fig. .

## Multi-Objective Recommendation

Recommendation with multiple objectives is a significant but challenging problem, with the core difficulty stemming from the potential conflicts between objective. In most real-world recommendation systems, recommendation accuracy (e.g., CTR-oriented objectives) is the dominating factor, while some studies believed that other characteristics, such as usability, profitability, usefulness, or diversity should be considered at the same time . When multiple objectives are concerned, it is expected to get a Pareto optimal/efficient recommendation .

The approaches on recommendation with multiple objectives to achieve Pareto efficiency can be categorized into two groups: evolutionary algorithm  and scalarization . jointly considered multiple trained recommendation algorithms with a Pareto-efficient manner, and conducted an evolutionary algorithm to find the appropriate parameters for weighted model combination. Besides, optimized GMV and CTR in e-commerce simultaneously based on multiple-gradient descent algorithm, which combines scalarization with Pareto-efficient SGD, and used a relaxed KKT condition. Our proposed method, MoFIR, belongs to scalarization, however, compared with earlier attempts in multi-objective recommendation , our method learns to adapt a single network for all the trade-off combinations of the inputted preference vectors, therefore it is able to approximate all solutions of the Pareto frontier after a single optimization run.

## RL for Recommendation

RL-based recommenders have recently become an important and attractive topic, as it is natural to model the recommendation process as a Markov Decision Process (MDP) and use RL agents to capture the dynamics in recommendation scenarios . Generally speaking, RL-based recommendation systems can be further classified into two categories: *policy-based* or *value-based* methods. On one hand, policy-based methods aim to learn strategies that generate actions based on state (such as recommending items). These methods are optimized by policy gradient, which can be deterministic approaches or stochastic approaches. On the other hand, value-based methods aims to model the quality (e.g. Q-value) of actions so that the best action corresponds to the one with the highest Q-value. Apart from using RL in general recommendation task, there also existed several work focusing on using RL in explainable recommendation through knowledge graphs .

Currently, there are very few studies using MORL in recommendation. studied multi-objective recommendation to capture users' objective-level preferences. However, unlike our proposed MoFIR, which learns a single parametric representation for optimal recommendation policies, they conducted a Pareto-oriented RL to generate the personalized objective weights in scalarization for each user, which is totally a different problem formulation.

# Preliminary

## Markov Decision Processes

In reinforcement learning, agents aim at learning to act in an environment in order to maximize their cumulative reward. A popular model for such problems is Markov Decision Processes (MDP), which is a tuple $M = (\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \mu, \gamma)$, where $S$ is a set of $n$ states, $\mathcal{A}$ is a set of $m$ actions, $\mathcal{P}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow [0,1]$ denotes the transition probability function, $\mathcal{R}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow \mathbb{R}$ is the reward function, $\mu: \mathcal{S} \rightarrow [0,1]$ is the starting state distribution, and $\gamma \in [0,1)$ is the discount factor. We denote the set of all stationary policies by $\Pi$, where a stationary policy $\pi \in \Pi: \mathcal{S} \rightarrow P(\mathcal{A})$ is a map from states to probability distributions over actions, with $\pi(a | s)$ denoting the probability of selecting action $a$ in state $s$. We aim to learn a policy $\pi \in \Pi$, able to maximize a performance measure, $J(\pi)$, which is typically taken to be the infinite horizon discounted total return, $$\label{eq:discounted_return}
\small
    J(\pi) \doteq \underset{\tau \sim \pi}{\mathrm{E}}\left[\sum_{t=0}^{\infty} \gamma^{\top} R\left(s_{t}, a_{t}, s_{t+1}\right)\right],$$ where $\tau$ denotes a trajectory, e.g., $\tau=(s_{0}, a_{0}, s_{1}, a_{1}, \dots)$, and $\tau \sim \pi$ indicates that the distribution over trajectories depends on $\pi$ : $s_{0} \sim \mu, a_{t} \sim \pi\left(\cdot | s_{t}\right), s_{t+1} \sim P\left(\cdot | s_{t}, a_{t}\right)$. We denote $R(\tau)$ as the discounted rewards of a trajectory, the on-policy value function as $V^{\pi}(s) \doteq$ $\mathrm{E}_{\tau \sim \pi}\left[R(\tau) | s_{0}=s\right]$, the on-policy action-value function as $Q^{\pi}(s, a) \doteq \mathrm{E}_{\tau \sim \pi}\left[R(\tau) | s_{0}=s, a_{0}=a\right]$, and the advantage function as $A^{\pi}(s, a) \doteq Q^{\pi}(s, a)-V^{\pi}(s)$.

## Multi-Objective Markov Decision Processes

Multi-Objective Markov Decision Processes (MOMDP) are MDPs with a vector-valued reward function $\textbf{r}_t=\textbf{R}(s_t,a_t)$, where each component of $\textbf{r}_t$ corresponds to one certain objective. A scalarization function *f* maps the multi-objective value of a policy $\pi$ to a scalar value. In this work, we consider the class of MOMDPs with linear preference functions, e.g., $\textit{f}_{\bm{\omega}}(\textbf{R}(s,a))=\bm{\omega}\textbf{R}(s,a)$. It is worth noting that if $\bm{\omega}$ is fixed to a single value, this MOMDP collapses into a standard MDP. An optimal solution for an MOMDP under linear *f* is a convex coverage set (CCS), e.g., a set of undominated policies containing at least one optimal policy for any linear scalarization.

## Conditioned Network

studied multi-objective reinforcement learning with linear preferences and proposed a novel algorithm for learning a single Q-network that is optimized over the entire space of preferences in a domain. The main idea is called Conditioned Network (CN), in which a Q-Network is augmented to output weight-dependent multi-objective Q-value-vectors, as is shown in Fig. (Conditioned Critic Network). Besides, to promote quick convergence on the new weight vector's policy and to maintain previously learned policies, the authors updated each experience tuple in a mini-batch w.r.t. the current weight vector and a random previously encountered weight vector. Specially, given a mini-batch of trajectories, they computed the loss for a given trajectory $(s_j,a_j,\textbf{r}_j,s_{j+1})$ as the sum of the loss on the active weight vector $\bm{\omega}_t$ and on $\bm{\omega}_j$ randomly sampled from the set of encountered weights. $$\label{eq:cn}
\begin{aligned}
&\frac{1}{2}\left[\left|\mathbf{y}_{\bm{\omega}_{t}}^{(j)}-\mathbf{Q}_{C N}\left(a_{j}, s_{j} ; \bm{\omega}_t\right)\right|+\left|\mathbf{y}_{\bm{\omega}_{j}}^{(j)}-\mathbf{Q}_{C N}\left(a_{j}, s_{j} ; \bm{\omega}_j\right)\right|\right]
\end{aligned}$$

$$\begin{aligned}
\textbf{y}_{\bm{\omega}}^{(j)}=\textbf{r}_{j}+\gamma \textbf{Q}_{C N}^{-}\left(\underset{a \in A}{\operatorname{argmax}} \textbf{Q}_{C N}\left(a, s_{j+1} ;\bm{\omega}\right) \cdot \bm{\omega}, s_{j+1} ; \bm{\omega}\right)
\end{aligned}$$ where $\textbf{Q}_{C N}\left(a, s ;\bm{\omega}\right)$ is the network's Q-value-vector for action $a$ in state $s$ and with weight vector $\bm{\omega}$. They claimed that training the same sample on two different weight vectors has the added advantage of forcing the network to identify that different weight vectors can have different Q-values for the same state.

In the original paper, the authors only proposed an algorithm based on Double DQN with discrete action space, which is not suitable for recommendation scenarios as the action space of recommendation is very large. Therefore, inspired by , we modify the traditional DDPG by introducing conditioned network into its policy network as well as critic network, and more importantly, we modify the original loss functions for both of them. More details about our modification will be introduced in Section .

# Problem Formulation

## **MOMDP for Recommendation**

At each timestamp ($t_1$, $t_2$, $t_3$, $t_4$, $t_5$, $\dots$), the recommendation agent $G$ will take the feature representation of the current user and item candidates $\mathcal{I}$ as input, and generate a list of items $L\in\mathcal{I}^K$ to recommend, where $K \geq 1$ after a user sends a request to it. User $u$ who has received the list of recommended item/items $L$ will give his/her feedback $B$ by his/her clicks on this set of items, which can be used to measure the recommendation performance. Besides, based on the recommendation results, we will acquire the total number of exposure for each item group, which can later be used to measure fairness. Thus, the state $s$ can be represented by user features (e.g., user's recent click history), action $a$ is represented by items in $L$, reward $\textbf{r}$ is the immediate reward vector after taking action $a$, with each component of $\textbf{r}$ corresponds to one certain objective (e.g., whether user clicks on an item in $L$ for utility objective or whether an item comes from predefined disadvantageous group for fairness objective). The problem formulation is formally presented as follows:

-   **State $\mathcal{S}$:** A state $s_t$ is the representation of user's most recent positive interaction history $H_t$ with the recommendation system, together with his/her demographic information (if exists).

-   **Action $\mathcal{A}$:** An action $a_t\ = \{ a_t^{1},\ \dots,\ a_t^{K}\}$ is a recommendation list with $K$ items to a user $u$ at time $t$ with current state $s_t$.

-   **Vector Reward Function $\bm{r}$:** A vector-valued reward function $\textbf{r}_t=\textbf{R}(s_t,a_t)$, where each component of $\textbf{r}_t$ corresponds to one certain objective. In this work, the reward vector includes two elements: utility objective and fairness objective. The details of the definition of our task-specific objectives will be introduced in the following section.

-   **Scalarization function** $\textit{f}$ **:** In this paper, we consider the class of MOMDPs with linear preferences functions $\textit{f}$, each objective is given a weight $\omega_i$, such that the scalarization function becomes $\textit{f}_{\bm{\omega}}(\textbf{R})=\bm{\omega} \cdot \textbf{R}$, where each $\omega_i \in [0,1]$ and $\sum_{i} \omega_i = 1$.

-   **Discount rate $\mathcal{\gamma}$:** $\mathcal{\gamma} \in [0,1]$ is a discount factor measuring the present value of long-term rewards.

We aim to learn a policy $\pi$, mapping from states to actions, to generate recommendations that achieve the Pareto efficient trade-off between fairness and utility.

## Multi-Objectives in Fair Recommendation

The reward vector is designed to measure the recommendation system's gain regarding to utility and fairness. While it is capable for our method to deal with multiple objectives simultaneously, for simplicity, we deliberately select click through rate and item (group) exposure fairness as show cases and call those two objectives recommendation utility and item exposure fairness respectively.

### Utility Objective

On one hand, given the recommendation based on the action $a_t$ and the user state $s_t$, the user will provide his/her feedback, e.g., click or purchase, etc. The recommender receives immediate reward $R_u(s_t, a_t)$ according to the user's positive feedback. We also normalize the reward value by dividing $K$, which is the length of the recommendation list.

$$\label{eq:reward}
\centering
\small
\begin{aligned}
    R_u(s_{t},a_{t}, s_{t+1}) &= \frac{\sum_{l=1}^{K} \vmathbb{1} (a_{t}^l \text{ gets positive feedback})}{K}\\
\end{aligned}$$

### Fairness Objective

On the other hand, based on the recommendation list $a_t$, the total number of exposure of each item group will be counted and used to measure exposure fairness. Here, we calculate the ratio of items from sensitive group to the total number of recommended items, and use a hinge loss with margin $\beta$ to punish the abuse of fairness. Usually, we set $\beta$ to be the ratio of the number of items in sensitive group to the total number of items.

$$\label{eq:cost}
\small
\centering
\begin{aligned}
    R_f(s_{t},a_{t}, s_{t+1}) &=  \max \left( \frac{\sum_{l=1}^{K} \vmathbb{1}(a_{t}^l\ is\ in\ sensitive\ group)}{K}, \beta  \right)
\end{aligned}$$

# Proposed Framework

## The Conditioned Actor

The conditioned actor is almost the same as traditional actor except that we condition the predictions of the policy network to the preference vectors. Practically, we concatenate the state representation $s_t$ with the vector $\bm{\omega}$ and train a neural network on this joint feature space, which is depicted in Fig. (Conditioned Actor Network). The conditioned actor $\pi$ parameterized by $\theta^\pi$ serves as a stochastic policy that samples an action $a_t\in\mathcal{I}^K$ given the current state $s_t\in\mathbb{R}^m$ of a user and the preference vector $\bm{\omega}$.

First on all, we define $s_t$ as the concatenation of the user embedding $\mathbf{e}_u\in\mathbb{R}^d$ and his/her recent history embedding $\mathbf{h}_u$: $$\label{eq:state_rep}
% s_t = [\bm{v}_u, h(H_t|u)]\label{eq:state_rep}
s_t = [\mathbf{e}_u; \mathbf{h}_u],$$ where the recent history embedding $\mathbf{h}_u=\mathrm{GRU}(H_t)$ is acquired by encoding $N$ item embeddings via Gated Recurrent Unites (GRU) , and $H_t = \{H_t^1, H_t^2, \dots, H_t^N\}$ denotes the most recent $N$ items from user $u$'s interaction history. We define the user's recent history is organized as a queue with fixed length, and update it only if the recommended item $a_t^l \in a_t$ receives a positive feedback, which ensures that the state can always represent the user's most recent interests. $$\small
    H_{t+1}=\left\{
    \begin{array}{cc}
         \{H_t^2,\ \dots,\ H_t^N,\ a_t^l\} & a_t^l \text{ gets positive feedback} \\
         H_t & \text{Otherwise}
    \end{array}
    \right.$$

Secondly, we assume that the probability of actions conditioned on states and preferences follows a continuous high-dimensional Gaussian distribution. We also assume it has mean $\mu\in\mathbb{R}^{Kd}$ and covariance matrix $\Sigma\in\mathbb{R}^{Kd\times Kd}$ (only elements at diagonal are non-zeros and there are actually $Kd$ parameters). In order to achieve better representation ability, we approximate the distribution via a deep neural network, which maps the encoded state $s_t$ and preferences $\bm{\omega}$ to $\mu$ and $\Sigma$. Specifically, we adopt a Multi Layer Perceptron (MLP) with tanh($\cdot$) as the non-linear activation function, $$(\mu,\Sigma)=\mathrm{MLP}(s_t,\bm{\omega}).$$ Once received $\mu$ and $\Sigma$, we sample a vector from the acquired Gaussian distribution $\mathcal{N}(\mu,\Sigma)$ and convert it into a proposal matrix $W\sim \mathcal{N}(\mu,\Sigma)\in\mathbb{R}^{K\times d}$, whose $k$-th row, denoted by $W_k\in\mathbb{R}^d$, represents an "ideal" embedding of a virtual item.

Finally, the probability matrix $P\in\mathbb{R}^{K\times |\mathcal{I}|}$ of selecting the $k$-th candidate item is given by $\small
    P_k = \mathrm{softmax}(W_k \mathcal{V}^\top),~ k=1,\ldots,K,$ where $\mathcal{V}\in\mathbb{R}^{|\mathcal{I}|\times d}$ is the embedding matrix of all candidate items. This is equivalent to using dot product to determine similarity between $W_k$ and any item. As the result of taking the action at step $t$, the actor recommends the $k$-th item as follows: $$\label{eq:action}
\small
    a_t^k = \mathop{\mathrm{arg\,max}}_{i\in \{1,\dots,|\mathcal{I}|\}} P_{k,i},~ \forall k=1,\ldots,K,
% \begin{cases}
%     \argmax_i P_{k,i} \forall k\in\{1,2,\dots,K\} & \text{if recommend}\\
%     \mathrm{Bernoulli}_i(P_{k,i}) & \text{if exploration}
% \end{cases}$$ where $P_{k,i}$ denotes the probability of taking the $i$-th item at rank $k$.

## The Conditioned Critic

The conditioned critic $\mu$ is also almost the same as traditional critic except that we concatenate the state representation $s_t$ with the vector $\bm{\omega}$ as well as the embedding of action $a_t$, and require the output to be a Q-value-vector with the size equal to the number of objectives, which is depicted in Fig. (Conditioned Critic Network). The conditioned critic $\mu$ is parameterized with $\theta^\mu$ and is constructed to approximate the true state-action value vector function $\textbf{Q}^\pi(s_t,a_t,\bm{\omega})$ and be used in the optimization of the actor. Following Eq. introduced in conditioned network , the conditioned critic network is updated according to temporal-difference learning that minimizes the following loss function: $$\label{eq:value_update}
\small
    \mathcal{L}(\theta^\mu) = \mathbb{E}_{s, a, \boldsymbol{\omega}}\left[\|\textbf{y}_t-\textbf{Q}_t\left(s, a, \bm{\omega}; \theta^\mu \right) \|_{2}^{2}\right]$$ where $\textbf{y}_t = \textbf{r}_t + \gamma \textbf{Q}_{\omega}(s_{t+1},a_{t+1},\bm{\omega};\theta^\mu)$.

## Parameters Training Procedure of MoFIR

We present the detailed training procedure of our proposed model, MoFIR, in Algorithm and the model architecture in Fig. . As mentioned before, we modify traditional single-objective DDPG into multi-objective DDPG by introducing the conditioned networks to both its actor network and critic network. In each episode, there are two phases — the trajectory generation phase (line 15-20) and model updating phase (line 22-32). In the trajectory generation phase, we sample one linear preference $\bm{\omega}_0$ and fix it to generate user-item interaction trajectories. Then in the model updating phase, we sample another $\mathcal{N}_\omega$ preferences together with $\bm{\omega}_0$ to update the conditioned actor network and the conditioned critic network. Here, we do not follow the original setting in , which only use one more random sampled preference vector, as observed that increasing the number of sampled preference vectors can improve the coverage ratio of RL agent and diminish the adaptation error.

# Experiments

## Dataset Description

```latex
\begin{adjustbox}{max width=\linewidth}
\begin{tabular}
    {lcccccccc} \toprule
    Dataset & \#users & \#items & \#act./user & \#act./item & \#act. & density \\\midrule
    Movielens100K & 943 & 1682 & 106 & 59.45 & 100,000 & 6.305\%\\
    Ciao & 2248 & 16861 & 16 & 2 & 36065 & 0.095\%\\
    Etsy & 1030 & 945 & 47 & 51 &  48080 & 4.940\%\\\bottomrule
\end{tabular}
\end{adjustbox}
```

To evaluate the models under different data scales, data sparsity and application scenarios, we perform experiments on three real-world dataset. Some basic statistics of the experimental datasets are shown in Table .

-   **Movielens:** We choose Movielens100K [^1], which includes about one hundred thousand user transactions, respectively (user id, item id, rating, timestamp, etc.).

-   **Ciao:** Ciao was collected by from a popular product review site, Epinions, in the month of May, 2011[^2]. For each user, they collected his/her profile, his/her ratings and his/her trust relations. For each rating, they collected the product name and its category, the rating score, the time point when the rating is created, and the helpfulness of this rating.

-   **Etsy:** We collect three weeks' user-item interaction data on a famous e-commerce platform, Etsy, from 7/1/2021 to 7/21/2021. For each record, we collect user id, item id and timestamp. Since the original data is very sparse, we filter out users and items with fewer than twenty interactions.

For each dataset, we first sort the records of each user based on the timestamp, and then split the records into training and testing sets chronologically by 4:1. The last item in the training set of each user is put into the validation set. Since we focus on item exposure fairness, we need to split items into two groups $G_0$ and $G_1$ based on item popularity. It would be desirable if we have the item impression/listing information and use it to group items, however, since Movielens and Ciao are public dataset and only have interaction data, we use the number of interaction to group items in them. Specifically, for Movielens and Ciao, the top 20% items in terms of number of interactions belong to the popular group $G_0$, and the remaining 80% belong to the long-tail group $G_1$, while for Etsy data, we additionally collect the number of listings/impressions per month for each item and group items based on this.

Moreover, for RL-based methods, we set the initial state for each user during training as the first five clicked items in the training set, and the initial state during testing as the last five clicked items in the training set. We also set the RL agent recommend ten items to a user each time.

```latex
\begin{table*}[]\caption{Summary of the performance on three datasets.
We evaluate for ranking ($Recall$, $F_1$ and $NDCG$, in percentage (\%) values, \% symbol is omitted in the table for clarity) and fairness ($Gini$ $Index$ and $Popularity$ $Rate$, also in \% values), whiles $K$ is the length of recommendation list.
When MoFIR is the best, its improvements against the best baseline are significant at p < 0.01.}
\centering
\begin{adjustbox}{max width=\linewidth}
\setlength{\tabcolsep}{7pt}
\begin{tabular}
    {m{1.53cm} ccc ccc ccc ccc ccc} \toprule
    \multirow{2}{*}{Methods}
    % & Metrics
    & \multicolumn{3}{c}{Recall (\%) $\uparrow$}
    & \multicolumn{3}{c}{F1 (\%) $\uparrow$}
    & \multicolumn{3}{c}{NDCG (\%) $\uparrow$}
    & \multicolumn{3}{c}{Gini Index (\%) $\downarrow$}
    & \multicolumn{3}{c}{Popularity Rate (\%) $\uparrow$}\\\cmidrule(lr){2-4} \cmidrule(lr){5-7} \cmidrule(lr){8-10} \cmidrule(lr){11-13} \cmidrule(lr){14-16}
 & K=5 & K=10 & K=20 & K=5 & K=10 & K=20 & K=5 & K=10 & K=20 & K=5 & K=10 & K=20 & K=5 & K=10 & K=20 \\\midrule
 \multicolumn{16}{c}{Movielens-100K} \\\midrule
MF  &  1.422 & 2.713 & 5.228 & 2.019 & 3.016 & 4.127 & 3.561 & 3.830 & 4.705 & 98.169 & 97.338 & 95.669 & 99.745 & 99.258 & 98.224 \\
BPR-MF  & 1.304 & 3.539 & 8.093 & 1.824 & 3.592 & 5.409 & 3.025 & 3.946 & 5.787 & 98.743 & 98.172 & 97.012 & 99.873 & 99.873 & 99.777 \\
NGCF  & 1.995 & 3.831 & 6.983 & 2.846 & 4.267 & 5.383 & 5.319 & 5.660 & 6.510 & 99.700 & 99.385 & 98.805 & 100.000 & 100.000 & 100.000 \\
LIRD  & 2.798 & 6.586 & 13.711 & 3.198 & 4.850 & 5.855 & 4.583 & 6.217 & 8.840 & 99.530 & 99.018 & 98.311 & 97.434 & 95.058 & 92.121 \\
\midrule

MF-FOE & 1.164 & 2.247 & 4.179 & 1.739 & 2.730 & 3.794 & 3.520 & 3.796 & 4.367 & 86.290 & 84.049 & 82.977 & 92.895 & 91.888 & 90.981 \\
BPR-FOE & 0.974 & 2.053 & 4.404 & 1.496 & 2.568 & 3.933 & 3.127 & 3.514 & 4.332 & 86.499 & 84.379 & 83.780 & 92.174 & 91.357 & 90.700 \\
NGCF-FOE & 1.193 & 1.987 & 4.251 & 1.759 & 2.398 & 3.698 & 4.033 & 3.897 & 4.633 & 96.919 & 94.525 & 90.436 & 100.000 & 100.000 & 100.000 \\\midrule

MF-MFR & 1.546 & 2.807 & 5.422 & 2.019 & 3.016 & 4.127 & 3.276 & 3.613 & 4.571 & 94.960 & 93.269 & 90.670 & 74.867 & 73.743 & 70.419 \\
BPR-MFR & 1.418 & 2.811 & 6.155 & 2.019 & 3.016 & 4.127 & 3.522 & 3.822 & 5.047 & 92.488 & 90.530 & 88.030 & 90.117 & 88.017 & 84.205 \\
NGCF-MFR & 1.456 & 2.900 & 6.570 & 0.395 & 0.687 & 0.599 & 3.041 & 3.472 & 4.928 & 94.712 & 93.487 & 91.538 & 97.794 & 96.352 & 93.674 \\\midrule

MoFIR-1.0 & \textbf{6.580} & \textbf{12.753} & \textbf{22.843} & \textbf{5.658} & \textbf{7.178} & \textbf{7.858} & \textbf{8.026} & \textbf{10.848} & \textbf{14.683} & 99.619 & 99.266 & 98.584 & 100.000 & 100.000 & 100.000\\
MoFIR-0.5 & 4.679 & 9.520 & 19.918 & 4.438 & 5.808 & 7.421 & 6.633 & 9.300 & 14.370 & 98.007 & 97.004 & 95.036 & 91.580 & 90.954 & 89.433 \\
MoFIR-0.1  & 1.958 & 4.199 & 7.083 & 5.658 & 7.178 & 7.858 & 2.827 & 3.558 & 4.687 & 95.665 & 93.647 & 89.698 & \textbf{49.608} & \textbf{48.759} & \textbf{47.587} \\
\midrule

\multicolumn{16}{c}{Ciao} \\\midrule
MF  &  0.518 & 1.938 & 3.100 & 0.395 & 0.687 & 0.599 & 0.408 & 0.924 & 1.264 & 78.154 & 71.322 & 62.380 & 69.088 & 63.835 & 57.098 \\
BPR-MF  & 1.087 & 2.204 & 4.607 & 0.677 & 0.770 & 0.858 & 0.776 & 1.181 & 1.900 & 83.717 & 78.214 & 71.702 & 79.826 & 74.949 & 69.580 \\
NGCF & 1.721 & 2.816 & 4.380 & 1.056 & 0.958 & 0.783 & 1.670 & 2.027 & 2.450 & 98.219 & 96.863 & 94.940 & 85.181 & 73.792 & 61.693 \\
LIRD & 0.766 & 2.448 & 3.599 & 0.554 & 1.082 & 0.921 & 1.393 & 2.638 & 3.277 & 99.364 & 98.771 & 97.632 & 63.936 & 76.223 & 63.632 \\\midrule

MF-FOE & 0.685 & 1.208 & 1.914 & 0.458 & 0.474 & 0.396 & 0.475 & 0.669 & 0.864 & 48.059 & 35.162 & 29.204 & 43.068 & 37.033 & 33.915 \\
BPR-FOE & 1.442 & 2.111 & 3.693 & 0.812 & 0.663 & 0.731 & 0.934 & 1.154 & 1.657 & 65.354 & 59.959 & 55.883 & 60.347 & 56.686 & 53.987 \\
NGCF-FOE & 1.234 & 1.907 & 2.903 & 0.651 & 0.583 & 0.566 & 0.937 & 1.156 & 1.477 & 79.313 & 74.038 & 71.335 & 43.357 & 34.226 & 30.391 \\\midrule

MF-MFR & 0.307 & 0.619 & 1.281 & 0.395 & 0.687 & 0.599 & 0.237 & 0.345 & 0.535 & 54.259 & 44.859 & 35.453 & 18.003 & 18.553 & 18.784 \\
BPR-MFR & 1.146 & 1.962 & 2.667 & 0.395 & 0.687 & 0.599 & 1.011 & 1.314 & 1.534 & 63.774 & 54.018 & 44.507 & 30.304 & 27.829 & 25.868 \\
NGCF-MFR & 1.284 & 2.131 & 4.033 & 2.019 & 3.016 & 4.127 & 1.014 & 1.342 & 1.901 & 98.387 & 97.519 & 96.002 & 52.388 & 43.430 & 36.498 \\\midrule
% LIRD-FOE  &  6.783 & \underline{5.566} & \underline{4.129} & 15.94 & 19.45 & \underline{22.74}\\\midrule

MoFIR-1.0  &  \textbf{2.162} & \textbf{3.867} & \textbf{5.866} & \textbf{1.626} & \textbf{1.513} & \textbf{1.323} & \textbf{4.000} & \textbf{4.764} & \textbf{5.813} & 96.633 & 93.921 & 90.109 & 93.025 & 88.263 & 80.796 \\
MoFIR-0.5  & 1.254 & 2.665 & 4.122 & 0.845 & 0.971 & 0.879 & 2.031 & 2.724 & 3.490 & 86.463 & 80.166 & 72.812 & 42.663 & 38.278 & 34.305 \\
MoFIR-0.1 & 0.892 & 1.610 & 2.338 & 0.557 & 0.532 & 0.445 & 1.054 & 1.311 & 1.576 & 94.819 & 92.460 & 89.317 & \textbf{21.100} & \textbf{19.522} & \textbf{16.795} \\\midrule

\multicolumn{16}{c}{Etsy} \\\midrule
MF  &  2.693 & 5.581 & 10.348 & 2.917 & 4.176 & 4.912 & 3.438 & 4.671 & 6.681 & 95.940 & 94.515 & 92.089 & 94.491 & 94.452 & 93.797 \\
BPR-MF  & 3.113 & 5.850 & 11.704 & 3.309 & 4.320 & 5.385 & 3.700 & 4.880 & 7.341 & 95.660 & 94.296 & 91.786 & 92.687 & 92.085 & 90.849 \\
NGCF & 3.414 & 6.026 & 11.746 & 3.674 & 4.498 & 5.406 & 4.180 & 5.238 & 7.610 & 96.046 & 94.141 & 91.084 & 88.904 & 88.264 & 86.998 \\
LIRD & 7.163 & 12.176 & 24.056 & 4.158 & 4.493 & 4.967 & 6.587 & 9.289 & 13.833 & 98.966 & 98.274 & 96.738 & 97.847 & 95.597 & 90.145 \\\midrule

MF-FOE & 1.382 & 2.436 & 4.515 & 1.641 & 2.160 & 2.704 & 2.111 & 2.482 & 3.318 & 67.431 & 63.189 & 60.058 & 54.898 & 48.865 & 44.758 \\
BPR-FOE & 1.503 & 2.808 & 5.513 & 1.802 & 2.468 & 3.132 & 2.328 & 2.783 & 3.844 & 65.476 & 62.232 & 59.803 & 55.209 & 49.263 & 45.276 \\
NGCF-FOE & 1.601 & 2.799 & 4.885 & 1.884 & 2.428 & 2.821 & 2.305 & 2.769 & 3.608 & 61.567 & 57.433 & 55.400 & 50.068 & 45.965 & 42.905 \\\midrule

MF-MFR & 2.482 & 5.150 & 10.279 & 2.917 & 4.176 & 4.912 & 3.265 & 4.504 & 6.671 & 94.564 & 92.155 & 87.365 & 91.620 & 88.565 & 83.497 \\
BPR-MFR & 2.510 & 4.849 & 9.711 & 2.917 & 4.176 & 4.912 & 3.144 & 4.110 & 6.206 & 93.410 & 90.616 & 85.817 & 83.899 & 80.165 & 73.031 \\
NGCF-MFR & 2.325 & 4.146 & 7.946 & 2.600 & 3.203 & 3.817 & 2.994 & 3.636 & 5.210 & 86.744 & 83.073 & 77.860 & 75.907 & 72.270 & 66.901 \\\midrule
% LIRD-FOE  &  6.783 & \underline{5.566} & \underline{4.129} & 15.94 & 19.45 & \underline{22.74}\\\midrule

MoFIR-1.0  &  6.690 & \textbf{13.87}1 & \textbf{24.72}8 & \textbf{4.833} & \textbf{5.932} & \textbf{6.238} & \textbf{9.183} & \textbf{13.629} & \textbf{19.822} & 99.133 & 98.381 & 96.821 & 98.293 & 98.041 & 97.861 \\
MoFIR-0.5  & 6.919 & 13.825 & 27.706 & 4.956 & 5.784 & 6.574 & 8.719 & 12.834 & 19.916 & 97.698 & 96.277 & 93.431 & 87.837 & 87.051 & 86.251 \\
MoFIR-0.1 & 1.340 & 2.966 & 5.864 & 1.151 & 1.641 & 1.895 & 1.778 & 2.839 & 4.425 & 97.231 & 95.670 & 93.195 & \textbf{23.628} & \textbf{23.550} & \textbf{23.016} \\\bottomrule
% MoFIR-0.1  &  \textbf{5.596} & \textbf{8.762} & \textbf{14.06} & \textbf{4.947} & \textbf{5.379} & \textbf{5.542} & \textbf{6.377} & \textbf{7.405} & \textbf{9.314} & 99.61 & 99.34 & 98.65 & 99.91 & 98.61 & 90.32\\\bottomrule
\end{tabular}\label{tab:result}
\end{adjustbox}
\vspace{-10pt}
\end{table*}
```

## Experimental Setup

### **Baselines:**

We compare our proposed method with the following baselines, including both traditional and reinforcement learning based recommendation models.

-   **MF**: Collaborative Filtering based on matrix factorization is a representative method for rating prediction. However, since not all dataset contains rating scores, we turn the rating prediction task into ranking prediction. Specially, the user and item interaction vectors are considered as the representation vector for each user and item.

-   **BPR-MF**: Bayesian Personalized Ranking is one of the most widely used ranking methods for top-K recommendation, which models recommendation as a pair-wise ranking problem. In the implementation, we conduct balanced negative sampling on unpurchased items for model learning.

-   **NGCF**: Neural Graph Collaborative Filtering is a neural network-based recommendation algorithm, which integrates the user-item interactions into the embedding learning process and exploits the graph structure by propagating embeddings on it to model the high-order connectivity.

-   **LIRD**: It is the short for List-wise Recommendation based on Deep reinforcement learning . The original paper utilized the concatenation of item embeddings to represent the user state, and the actor will provide a list of K items as action.

We also include two state-of-the-art fairness framework to show the effectiveness of our proposed method.

-   **FOE**: Fairness of Exposure in Ranking (FOE) is a type of post-processing algorithm incorporating a standard linear program and the Birkhoff-von Neumann decomposition. It is originally designed for searching problems, so we follow the same modification method mentioned in , and use ranking prediction model such as MF, BPR, and NGCF as the base ranker, where the raw utility is given by the predicted probability of user $i$ clicking item $j$. In our experiment, we have **MF-FOE**, **BPR-FOE** and **NGCF-FOE** as our fairness baselines.

-   **MFR**: Multi-FR (MFR) is a generic fairness-aware recommendation framework with multi-objective optimization, which jointly optimizes fairness and utility for two-sided recommendation. In our experiment, we only choose its item popular fairness. We also modify it as the original fairness considers position bias as well, which is not the same setting as ours. Finally, we have **MF-MFR**, **BPR-MFR** and **NGCF-MFR**.

We implement MF, BPR-MF, NGCF, MF-FOE, BPR-FOE, NGCF-FOE, MF-MFR BPR-MFR and NGCF-MFR using *Pytorch* with Adam optimizer. For all of them, we consider latent dimensions $d$ from {16, 32, 64, 128, 256}, learning rate $lr$ from {1e-1, 5e-2, 1e-2, …, 5e-4, 1e-4}, and the L2 penalty is chosen from {0.01, 0.1, 1}. We tune the hyper-parameters using the validation set and terminate training when the performance on the validation set does not change within 5 epochs. Similarly, we implement **MoFIR** with $Pytorch$. We first perform basic MF to pretrain 16-dimensional user and item embeddings, and fix them through training and test. We set $|H_t|=5$, and use 2 layer of GRU to get state representation $s_t$. For the actor network and the critic network, we use two hidden layer MLP with tanh($\cdot$) as activation function. Finally, we fine-tune MoFIR's hyper-parameters on our validation set. In order to examine the trade-off between performance and fairness, we use different level of preference vectors in test and denote the resulting alternatives as **MoFIR-1.0**, **MoFIR-0.5**, and **MoFIR-0.1**, where the scalar is the weight on the recommendation utility objective.

### **Evaluation Metrics:**

We select several most commonly used top-K ranking metrics to evaluate each model's recommendation performance, including **Recall**, **F1 Score**, and **NDCG**. Additionally, we include two fairness measures—**Gini Index** and **Popularity Rate**, with respect to measure item exposure fairness at an individual level and a group level, respectively. Given a list of impressions from all items, $\mathcal{M}=[g_{1},g_{2},...,g_{|\mathcal{I}|}]$, the Gini Index can be calculated as, $\small
    Gini\ Index(\mathcal{G}) = \frac{1}{2|\mathcal{I}|^2\bar{g}}\sum_{i=1}^{|\mathcal{I}|}\sum_{j=1}^{|\mathcal{I}|} |g_{i} - g_{j}|,$ where $\bar{g}$ represents the mean of all item impressions, while popularity Rate, simply refers to the ratio of the number of popular items in the recommendation list to the total number of items in the list. Besides, both of the two fairness measures are the smaller, the fairer to the recommender system.

## Experimental Results

The major experimental results are shown in Table , besides, we also plot the approximate Pareto frontier between NDCG and Long-tail Rate (namely, 1-Popularity Rate) in Fig. . We analyze and discuss the results in terms of the following perspectives.

### Recommendation Performance

For recommendation perfor-mance, we compare MoFIR-1.0 with MF, BPR, NGCF, and LIRD based on $Recall@k$, $F1@k$ and $NDCG@k$ and provide the these results of the recommendation performance in Table . Among all the baseline models, we can see that all sequential recommendation methods (LIRD, MoFIR-1.0) are much better than the traditional method, which demonstrates the superiority of sequential recommendation on top-K ranking tasks. LIRD is the strongest baseline in all three datasets on all performance metrics: when averaging across recommendation lengths LIRD achieve

Our MoFIR approach achieves the best top-K recommendation performance against all baselines on both datasets. On the one hand, when averaging across recommendation lengths on Movielens100K, MoFIR gets 33.09% improvement than NGCF; on the other hand, when averaging across recommendation lengths on Movielens1M, MoFIR gets 18.65 % improvement than LIRD. These observations imply that the proposed method does have the ability to capture dynamic user-item interactions, which captures better user preferences resulting in better recommendation results. Another interesting observation is that MoFIR is better than LIRD even though they use the same state representation and similar training procedure. This may be attributed to the trust-region-based optimization method, which stabilizes the model learning process.

### Fairness Performance

For fairness performance, we compare MoFIRs with FOE-based methods and MFR-based methods based on $Gini\ Index@k$ and $Popularity\ Rate@k$, which are also shown in Table . It is easy to find that there does exist a trade-off between the recommendation performance and the fairness performance, which is understandable, as most of the long-tail items have relatively fewer interactions with users. Besides, MoFIR is able adjust the degree of trade-off between utility and fairness through simply modifying the weight of recommendation utility objective. In Table , we can find that even MoFIR has the similar performance of fairness with other baseline, it can still achieve much better recommendation performance, for example, BPR-FOE and MoFIR-0.5 in Movielens100k or NGCF-FOE and MoFIR-0.5 in Ciao or MF-MFR and MoFIR-0.5 in Etsy. One disadvantage of current MoFIR is that it has a very high Gini Index even when the popularity rate is very small (e.g., MoFIR-0.1 in Ciao and Etsy). One possible reason could be . Taking one individual level fairness as additional objective should solve this problem.

### Fairness-Utility Trade-off

In order to better illustrate the trade-off between utility and fairness, we fix the length of the recommendation list at 20 and plot NDCG against Long-tail Rate in Fig. for all datasets, where the long-tail rate is equal to one minus popularity rate.

# Conclusion

In this work, we aim to achieve the Pareto Efficient trade-off between fairness and utility in recommendation systems and characterize their Pareto Frontier in the objective space in order to find solutions with different level of trade-off. We accomplish the task by proposing a fairness-aware recommendation framework using multi-objective reinforcement learning (MORL) with linear preferences, called MoFIR, which aims to learn a single parametric representation for optimal recommendation policies over the space of all possible preferences. Experiments across three different datasets demonstrate the effectiveness of our approach in both fairness measures and recommendation performance.

[^1]: <https://grouplens.org/datasets/Movielens/>

[^2]: https://www.cse.msu.edu/ tangjili/datasetcode/truststudy.htm