\section{Introduction}

\begin{figure}[hbt!]
\centering
\includegraphics[width=\columnwidth]{RSODD.png}
\caption{An example of a chatbot system that cares for senior citizens living alone. The utterance in red highlights the model's mistaken identity as a chef rather than the caring chatbot.}
\label{fig:intro}
\end{figure}


Recent large-scale language models (LMs) have brought numerous breakthroughs in open-domain dialogue systems, yielding human-like responses \cite{zhang-etal-2020-dialogpt,adiwardana2020towards,NEURIPS2020_1457c0d6,roller-etal-2021-recipes,kim-etal-2021-changes}.
In addition, there have been progresses in controlling dialogue systems in persona, style, and safety \cite{zhang-etal-2018-personalizing,smith2020controlling,xu-etal-2021-bot}, which impose consistency on chatbot's personality and mitigate undesirable features such as toxic or biased language.
However, building a chatbot system combining these capabilities is still challenging, which requires numerous human-human dialogues for those conversational skills.

Most task-oriented dialogue systems conduct specific roles such as booking assistants, information providers, customer service agents, or personal assistants \cite{eric-etal-2017-key,xu2017new,budzianowski-etal-2018-multiwoz}. 
However, studies on open-domain dialogue systems that perform specific roles have been insufficiently investigated, even though the role can be defined for the practical chatbot systems (e.g., chatbots that care for senior citizens living alone, or counseling chatbots).
In these cases, the chatbot systems do not have an explicit goal or task other than to proactively engage in conversations, but may have system policies on whether or not certain types of utterances are allowed (example in Figure \ref{fig:intro}).




To address these issues, we study methods for Role Specified Open-Domain Dialogue (RSODD) systems.
The goal of the system is conversing naturally with humans on open-ended topics while keeping conditions of given role.
Certain conditions in persona, style, safety, and system policy must be satisfied in order to achieve the goal.
We consider a general and scalable framework to treat them, instead of using individual approaches to control each.

In particular, we present a Human-AI collaborative data construction method to build a scalable supervisory dataset from scratch for role-satisfying open-domain dialogues (Figure~\ref{fig:process}).
We propose to leverage large-scale LMs for generating entire dialogue sessions between user and system by in-context few-shot learning manner \cite{NEURIPS2020_1457c0d6,kim-etal-2021-changes}, followed by human-interactive correction processes.
Our method can significantly reduce the cost of building dataset when compared to manually producing gold dialogues (Section \ref{sec:filter}).
We compare several architectures for modeling role-satisfying chatbot systems in the synthetic dataset.
In extensive experiments and ablation studies, we show that the proposed models considerably reduce undesirable utterances that violate the given role specification compared to the in-context learning baseline, while achieving competitive SSA \cite{adiwardana2020towards} scores for their responses.
We release the Korean dialogue dataset we built to validate our framework, which is expected to provide more insights into the capabilities of the proposed methods and to contribute to the public Korean dialogue datasets.

The contribution of our work is summarized as follows.
\begin{enumerate}
\item We make a step towards role specified open-domain dialogue (RSODD) systems which are capable of conversing naturally on open-ended topics while satisfying role specifications.
\item We suggest employing in-context learning of large-scale LMs as a scalable method for dialogue data construction.

\item We compare various architectures for RSODD systems to analyze the capabilities in terms of satisfying system policies.
\item We release the first Korean RSODD dataset while demonstrating the effectiveness of data construction method.
\end{enumerate}


\begin{figure}[t!]
\centering
\includegraphics[width=0.95\columnwidth]{process.png}
\caption{Our proposed framwork: (1) the dialogue developer provides a role specification of the desired chatbot and a few dialogue examples, (2) large-scale LMs generate entire dialogues and crowd workers filter the system's utterances, (3) a dialogue model is trained with supervised learning on the dataset, (4) crowd workers chat 1:1 with the chatbot and give additional feedback.}
\label{fig:process}
\end{figure}


\section{Related Work}
\paragraph{Pretrained LM in Open-domain dialogue}
Many prior works tried to pretrain the models on large-scale social comment chains data like Reddit to model conversational behavior~\cite{zhang-etal-2020-dialogpt, adiwardana2020towards}, followed by finetuning on the diverse target dialogue dataset to improve engagingness and humanness~\cite{roller-etal-2021-recipes}. To avoid undesired behaviors of the models including toxicity and bias from the human-human conversation, they merely exclude some parts of training data using automatic filtering by predefined criteria.

\paragraph{Synthetic Dialogue Generation}
To reduce cost of dialogue collection, there have been many approaches to generate synthetic dialogues~\cite{schatzmann-etal-2007-agenda, shah-etal-2018-bootstrapping, campagna-etal-2020-zero}. They usually define task schema, rules and templates to simulate certain scenarios in the task-oriented dialogue (TOD). \citet{kim-etal-2021-neuralwoz} proposed neural simulation approach using pretrained LMs for a fast domain adaptation in the TOD. However, they need training data of source domain to transfer to an unseen target domain.

\citet{xu-etal-2021-bot} proposed Bot-Adversarial Dialogue method to make existing models safer in terms of offensive or toxic behavior. 
\citet{sun-etal-2021-adding} extends existing task-oriented dialogue dataset to open-domain chit-chat using the pretrained LMs. 
Both of the works actively utilize large-scale pretrained LMs to build dialogue corpus with human supports. 
We also introduce human-AI collaborative dialogue collection method, while especially utilizing few-shot in-context learning ability of large-scale LM~\cite{NEURIPS2020_1457c0d6, kim-etal-2021-changes}.
To the best of our knowledge, this work is the first to propose using the in-context learning approach
to generate synthetic samples from large-scale language models for the purpose of \textit{dialogue} data generation.

\paragraph{On the Role in Dialogue} In TOD, the system side plays functional roles utilizing explicit knowledge base of specific domain~\cite{williams-etal-2013-dialog,henderson-etal-2014-second,henderson2014third, eric-etal-2017-key,xu2017new,budzianowski-etal-2018-multiwoz}. For example, agent in \citet{budzianowski-etal-2018-multiwoz} played booking assistant or information provider in various domain such as restaurant and hotel. 
On the other hand, \citet{zhang-etal-2018-personalizing} proposed assigning explicit persona to each dialogue agent, promoting the agent to make more specific and consistent responses in open-domain dialogue setting. 
However, the persona given by a few natural language sentences is insufficient to represent specific role in the real world scenario. \citet{sun-etal-2021-adding} also proposed guidelines of appropriate and inappropriate behaviors as a role of virtual assistant. 
We note that a recent concurrent work \cite{shuster2021me} studied conditioning dialogue models with similar motivations. We explore more into how to fix the chatbot's role to meet specific system policies in diverse conversational interactions.


\paragraph{Companion Dialogue System}
Building companionable dialogue system has long been investigated along with the advancement of open-domain dialogue models. \citet{webb-etal-2010-wizard} defines companions to be persistent, collaborative and conversational partners, and proposes evaluation strategies: empathy, positivity, and adaptive. \citet{Kopp2018ConversationalAF} introduced conversational assistants for elderly users which carry out socially cooperative dialogue. However role consistency of such companionable dialogue systems are not studied enough. %







\begin{figure}[t]
\centering
\includegraphics[width=\columnwidth]{generation_example.png}
\caption{An example of in-context one-shot dialogue generation for the data construction process. (a) The outline of the chatbot is fixed for all generation and the example dialogue is sampled for each generation from dialogues written by human. (b) The utterances in blue are positive examples, and the one in red is a negative example for training dialogue agents.}
\label{fig:generate}
\end{figure}

\section{Data Construction}
\label{sec:data}
In this section, we describe a framework to gather supervisory data for building RSODD systems.
The input to the framework is a role specification described by the chatbot developer (Table \ref{tab:spec} for example), which defines the conditions in the dialogue interactions for the system. 
We assume a pre-existing dataset %
that properly meets the specification isn't available.
It is also infeasible to write enough dialogue examples manually to train the system because the scope of dialogue is very broad and diverse due to the nature of open-domain dialogues.
To remedy this, we focus on composing the dataset with a few samples of human-written dialogues using in-context few-shot learning of large-scale LMs \cite{NEURIPS2020_1457c0d6,liu2021makes}.

\begin{table*}
\centering
\begin{adjustbox}{max width=\textwidth}
\begin{tabular}{l|l|l|l}
\toprule
\multicolumn{4}{l}{\textbf{Outline}} \\
\hline
\multicolumn{4}{l}{
\makecell[l]{The chatbot is an artificial intelligence agent that regularly calls and converses with senior citizens. \\
Initiate the conversation and react friendly to the user's utterances.  \\
Talk about everyday topics for 10-15 turns and end the call.}}\\
\midrule
\multicolumn{4}{l}{\textbf{Details}} \\
\hline
\multicolumn{2}{l|}{\textbf{Categories}} & \multicolumn{2}{l}{\textbf{Specification}} \\
\hline
\multicolumn{2}{l|}{Sensibleness} & Description & Speech that does not properly understand the context is restricted. \\
\hline
\multicolumn{2}{l|}{Style} & Description & Speech should be polite$^*$ and respectful.\\
\hline
\multicolumn{2}{l|}{Safety} & Description & Hate speech, toxic or biased language, and remarks containing personally identifiable information are all prohibited. \\
\hline
\multicolumn{2}{l|}{\multirow{2}{*}{Persona}} & Description & \makecell[l]{Keep the identity of an `AI chatbot that calls to the user.' \\
Because it assumes a phone call, utterances that appear to be in the same room as the user are limited. \\
Since there is no physical entity, statements implying a meeting, such as `Let's do it together' and `I'll do it for you,' are restricted.} \\
\cline{3-4}
\multicolumn{2}{l|}{} & Examples & \makecell[l]{"Grandma! I'm here!" (X) \\
"Would you like to walk with me?" (X) \\
"I'll invite you to my house later" (X)} \\
\hline
\multirow{4}{*}{System Policy} & \multirow{2}{*}{Temporality} & Description & \makecell[l]{
Because it is not given time-related information, the chatbot is unable to offer a timely utterance. \\
Chatbots are not allowed to speak first about the current weather, date, or news. \\
However, if the user brings up the subject first, it is feasible to agree.} \\
\cline{3-4}
& & Examples & \makecell[l]{
"Because the weather is turning cold these days, you should dress warmly." (X) \\
"Happy Holidays!" (X) \\
"Did you watch the baseball championship game today?" (X)
}\\
\cline{2-4}
& \multirow{2}{*}{Unsupported Features} & Description & \makecell[l]{
It does not provide any other functions other than making phone calls and chatting. \\
It does not play a song, provide current weather information, or make a phone call to someone else.
} \\
\cline{3-4}
& & Examples & \makecell[l]{
"I'll play a song." (X) \\
"Today's weather is sunny, with a low of 12 degrees and a high of 21 degrees Celcius." (X) \\
"Then I'll call your daughter." (X)
}\\
\bottomrule
\multicolumn{3}{l}{\footnotesize{$^*$ There are polite words and honorifics in the Korean language.}}
\end{tabular}
\end{adjustbox}
\caption{Example role specification used. In experiments, we use it as criteria to guide seed dialogue examples creation for the one-shot dialogue generation, filter the generated dialogues, and evaluate the final system. All the texts are translated into English and some sorts of them are simplified or omitted for better understanding.}
\label{tab:spec}
\end{table*}


\subsection{One-shot Dialogue Generation}
\label{sec:dialog}
As reported in \citet{kim-etal-2021-changes}, 
large-scale LMs can generate dialogues with a specific personality, given a prompt consisting of a brief description of the chatbot's properties and few dialogue examples.
We use this method to build the entire dataset.
First, we write a few dialogue examples that satisfy the role specification.
And we attach each of them at the end of the system description (Outline in Table \ref{tab:spec}) to compose input prompts for one-shot in-context learning.
Figure \ref{fig:generate} (a) shows an example input.
Then, the LM generates whole dialogue sessions. That is, the LM acts as both a system and a user (Figure \ref{fig:generate} (b)).
Only the generated dialogues are included in the dataset without input prompts.

\subsection{Human Filtering}
\label{sec:filter}
It is difficult to include all the details of specifications in the prompt and reflect them in the generation.
Therefore, we employ human annotation on the generated data.
We give the annotator each conversation session and ask them to label the point where the first out-of-bounds\footnote{An utterance that does not meet the conditions of the given role specification (Table \ref{tab:spec} for example).} occurred.
Figure \ref{fig:generate} (b) shows an example of a verified dialogue (more examples are provided in Appendix \ref{sec:ex}).
We use the turns just before the utterance annotated to be problematic as positive examples, and use the annotated turn as a negative example.
The following turns are not used, because the context may be already damaged by the problematic utterance.
Annotation time per dialogue session is about 88s, which is 13.3 times faster than human writing time per session (about 1170s).
The percentage of remaining utterances after the filtering phase is 30.4\% (See Table \ref{tab:stat}).

\subsection{Collecting Human-Bot Dialogues}
\label{sec:feedback}
Although human filtering is included in the dataset building process, the actual utterances are all machine-generated.
Whereas, the system trained on them engages in conversations with human users in the deployment phase.
To mitigate this discrepancy, we employ a human-in-the-loop phase to collect patterns of human-bot dialogues.
Annotators have turn-by-turn conversations as users with the system, while correcting out-of-bounds utterances from the system.
We incorporated LM's assistance into this process to help speed the task; %
if the system's response is not appropriate, an annotator presses the `Fix' button (Figure \ref{fig:ui} in Appendix showing the user interface) to call the large-scale LM to generate an alternative utterance.
The worker continues the conversation if the alternate utterance is appropriate, and if it is still not corrected, presses the `Fix' button repeatedly.
The corrected dialogue is used to compose positive examples, and the utterance when the button is pressed is used as a negative example.
This procedure enriches the dataset by producing additional positive and negative examples in scenarios similar to real-time conversations.


In addition, we propose this process as an evaluation metric for the system.
Since the action of pressing the `Fix' button means that an inappropriate utterance is returned from the system, it can be used for the system's \textbf{error rate}; the rate of the corrected responses among the total returned responses.
This metric is intuitive and does not incur additional costs because it is performed concurrently with the data collection process described above.



\section{Models}
\subsection{Notation}
Response prediction task in open-domain dialogues is predicting an utterance $y=\{y_1, y_2, \cdots, y_{|y|}\}$ given a dialogue history $x=\{s_1, u_1, s_2, u_2, \cdots, s_{k}, u_{k}\}$, where $s_i$ and $u_i$ are system utterance and user utterance respectively.



\subsection{Out-of-Bounds Detection}
The most straightforward method for constraining the system's utterances according to the role specification is to detect and discard out-of-bounds utterances.
We consider a BERT-based \cite{devlin-etal-2019-bert} binary classifier fine-tuned to classify positive/negative examples in datasets.
Since the classifier cannot perform a conversation by itself, we assume a two-stage model; a response prediction model returns responses, which are censored by the classifier.
If an out-of-bounds utterance is detected, we select and return one of several pre-defined questions about other topics, similar to the method used in \citet{xu-etal-2021-bot}. 
Instead of random choice, we selected the question with lowest PPL measured using LMs, as depicted in Section \ref{sec:retriever}.

\subsection{Response Selection}
\label{sec:retriever}
Another conceivable approach to constrain the system's utterances is to pre-filter the response candidates for response selection models.
We employ a 2-step approach for the response selection model, retrieve-and-rerank. 
The retriever of poly-encoder architecture \cite{humeau2019poly} rapidly finds the top-k plausible responses from the response candidates, which are then carefully reranked by the reranker of cross-encoder architecture.
Both retriever and reranker are fine-tuned in the same way as \citet{humeau2019poly} depicts.  


Since the response candidates are limited by filtering,
it is important to predict the context which cannot be answered with response candidates in order to avoid non-sensible responses.
One of the effective methods to predict unanswerable contexts is to utilize the uncertainty of the model \cite{feng-etal-2020-none, penha-hauff-2021-calibration}.
\citet{penha-hauff-2021-calibration} proposed a risk-aware score using MC Dropout \cite{gal2016dropout} and we employ a similar approach using thresholding; 
we score the retrieved responses using mean and variance of the predictive distribution from MC Dropout:
\begin{equation*}
S_{\text{D}}(x,\hat{y}) = E[R_{\hat{y}}] - var[R_{\hat{y}}],
\end{equation*}
where $\hat{y}$ is a candidate response that is retrieved, $R_{\hat{y}} = \{f(x,\hat{y}^1), f(x,\hat{y}^2),\cdots  f(x,\hat{y}^m)\}$ is a predictive distribution obtained by employing dropout \cite{srivastava2014dropout} at test time and conducting $m$ forward passes, and $f$ is a score function of reranker.
If all the scores of retrieved responses are lower than a certain threshold, it is predicted as unanswerable context. %

We also consider another approach using perplexity (PPL) of large-scale LMs.
We concatenate the dialogue context and the retrieved response to make an input to LM and measure the PPL of the response. 
Thresholding is employed for final decision and the threshold is determined on the validation set (See Appendix \ref{sec:valid}).

\begin{figure}[t]
\centering
\includegraphics[width=\columnwidth]{retrieve-fail-generate.png}
\caption{Retrieve-fail-Generate pipeline.}
\label{fig:retgen}
\end{figure}

\subsection{Response Generation}
Fine-tuning LMs on target data is known to be effective in learning desirable traits of focused tasks \cite{roller-etal-2021-recipes,gehman-etal-2020-realtoxicityprompts}.
Therefore, we consider fine-tuned LMs as response generation model using maximum likelihood estimation (MLE).
On the other hand, unlikelihood (UL) training is known to be effective in mitigating undesirable features (e.g., token repetition or logical inconsistency) of generative models \cite{li-etal-2020-dont,welleck2019neural}.
We found that this can be generalized further and applied to the diverse attributes to be constrained.
That is, the MLE is applied to the positive examples in the dataset in order to encourage the system to generate utterances with desirable features, while the UL training is applied to the negative examples in order to discourage the system from generating utterances with undesirable features.
Both types of training are performed concurrently. %

Formally, we fine-tune LMs as generative models using maximum likelihood estimation (MLE), which minimizes:
\begin{multline*}
\mathcal{L}_{\text{MLE}}^{n}(p_\theta, x^n, y^n) = - \sum_{t}{\log{p_\theta(y_t^n|x^n,y_{<t}^n)}},
\end{multline*}
where $x^n$ is a dialogue history in positive examples and $y^n$ is a corresponding gold response.
Unlikelihood training is done by adding a loss that penalizes the token set $C_t$ to be constrained,
\begin{multline*}
\mathcal{L}_{\text{UL}}^n(p_\theta, C_{1:T}, x, y) = \\
-\sum_{t}{\sum_{y_c\in C_t}{\log{(1-p_\theta(y_c|x, y_{<t}))}}},
\end{multline*}
where $C_t \subseteq \mathcal{V}$ is a subset of the vocabulary.
We employ this to the negative examples in dataset $\{(x^-, y^-)\}$.
For this, $C_t$ is defined as $\{y_t^-\}$, which results in the following:
\begin{multline*}
\mathcal{L}_{\text{UL}}^{-}(p_\theta, x^-, y^-) = \\
-\sum_{t}\log{(1-p_\theta(y_t^-|x, y_{<t}^-))}.
\end{multline*}
The final loss function consists of mixing MLE loss and UL loss,
\begin{equation}
\mathcal{L} = \mathcal{L}_{\text{MLE}}^+ + \alpha\mathcal{L_{\text{UL}}^-},
\end{equation}
where $\alpha \in \mathbb{R}$ is the mixing hyper-parameter.

\subsection{Retrieve-fail-Generate}
We also consider a pipelined approach that consists of response selection and generation models.
We first tried a Retrieve-and-Refine architecture \cite{roller-etal-2021-recipes, weston-etal-2018-retrieve}, but it failed in $\alpha$-blending\footnote{In our experiments, all retrieved responses are copied or ignored depending on the $\alpha$ value, reducing the model to a retriever or generator. This has also been highlighted in a recent concurrent study \cite{han2021understanding}.}.
In addition, according to \citet{roller-etal-2021-recipes}, the Retrieve-and-Refine strategy delivers marginal or no improvements over the generator.
Therefore, we build another pipeline, refered to as a Retrieve-fail-Generate model (Figure \ref{fig:retgen}). 
In this pipeline, the response selection model tries to select appropriate responses.
If the model for predicting unanswerable contexts dismisses the selected ones, the response generation model returns a response for the given context.
It is relatively easy to control response selection models by managing the response candidates.
Hence, the response selection models are responsible for majority of the responses, and the generation model is only used when the response selection fails.

\begin{table}
\centering
\begin{adjustbox}{max width=\columnwidth}
    \begin{threeparttable}
    \begin{tabular}{l|rrrr}
    \toprule
    \textbf{Dialogue Type} & \textbf{Example} & \textbf{Generated} & \textbf{Filtered}  & \textbf{Feedback}\\
    \midrule
    \# Dialogues          & 250       & 25,000    & 17,617    & 1,623 \\
    \# Turns             & 3,893     & 510,028   & 154,903   & 29,365 \\
    Avg. turns / dialogue    & 15.57       & 20.40    & 8.79    & 18.09 \\
    \# Pos. examples      & -         & -         & 47,091    & 10,829 \\
    \# Neg. examples      & -         & -         & 18,583    & 3,529 \\
    \# Unique sys-turns  & 1,805     & 170,527   & 36,227    & 9,405 \\
    \midrule
    \# Words          & 35,253       & 4,292,613    & 705,253    & 178,357 \\
    Avg. words / turn        & 9.06       & 8.42    & 4.55    & 6.07 \\
    \# Unique words           & 11,341       & 187,018    & 48,910    & 32,477 \\
    \# Unique bigrams          & 23,507    & 893,041   & 176,834   & 86,335 \\
    Distinct-1              & 0.3215    & 0.0436    & 0.0694    & 0.1821 \\
    Distinct-2              & 0.7907    & 0.2538    & 0.3067    & 0.5795 \\
    \bottomrule
    \end{tabular}
    \end{threeparttable}
\end{adjustbox}
\caption{Statistics of dataset collected in Section \ref{sec:dataset}. \textbf{Example} is a human-written dialogue set for in-context learning. \textbf{Generated} is a generated set by LMs (Section \ref{sec:dialog}). \textbf{Filtered} is a set after human filtering phase (Section \ref{sec:filter}). \textbf{Feedback} is human-bot dialogues with corrections (Section \ref{sec:feedback}). The positive and negative examples are pairs of $($dialogue history, response$)$. Distinct-1/2 \cite{li-etal-2016-diversity} is the number of distinct uni- or bi-grams divided by total number of words.}
\label{tab:stat}
\end{table}






\begin{table*}[t!]
\centering
\begin{adjustbox}{max width=\textwidth}
\begin{tabular}{l|cc|cc|ccccc}
\toprule
\multicolumn{1}{c}{}  & \multicolumn{2}{c}{\textbf{Automatic Metrics}} & \multicolumn{7}{c}{\textbf{Human Evaluations}}  \\
  \cmidrule(lr){2-3}\cmidrule(lr){4-10}
\multicolumn{6}{c}{}  & \textbf{User} & \multicolumn{3}{c}{\textbf{System}}  \\
\cmidrule(lr){7-7}\cmidrule(lr){8-10}
\textbf{Model} & \textbf{Distinct-1} & \textbf{Distinct-2} & \textbf{Fluency} & \textbf{Coherence} & \textbf{Situation} & \textbf{Persona} & \textbf{Persona} & \textbf{Style} & \textbf{Safety} \\
\midrule
1.3B & 0.2959 (0.0042) & 0.6630 (0.0053) & 4.98 (0.02) & 4.54 (0.21) & 4.57 (0.29)	& 4.54 (0.15) & 4.31 (0.23) & 4.91 (0.05) & 4.98 (0.03) \\
13B & 0.3075 (0.0037) & 0.6500 (0.0054) & 4.97 (0.02) & 4.55 (0.14) & 4.74 (0.23) & 4.65 (0.11) & 4.33 (0.20) & 4.93 (0.04) & 4.98 (0.02) \\
39B & 0.3334 (0.0038) & 0.6779 (0.0061) & 4.98 (0.03) & \textbf{4.59} (0.19) & 4.69 (0.22) & 4.69 (0.12) & 4.37 (0.21) & 4.88 (0.05) & 4.97 (0.02) \\
82B & \textbf{0.3402} (0.0040) & \textbf{0.7014} (0.0057) & 4.98 (0.02) & 4.56 (0.24) & \textbf{4.78} (0.17) & \textbf{4.74} (0.15) & \textbf{4.49} (0.17) & \textbf{4.96} (0.07) & 4.96 (0.03) \\
\bottomrule
\end{tabular}
\end{adjustbox}
\caption{Automated metric and human evaluations for generated dialogues from various size of LMs. Scores are averaged (standard deviation in brackets).}
\label{tab:dialog}
\end{table*}



\begin{table*}[t!]
\centering
\begin{adjustbox}{max width=\textwidth}
\begin{tabular}{lrr|rrrrr}
\toprule
\textbf{Model} & \textbf{\# of system turns} & \textbf{error rate} & \textbf{not sensible} & \textbf{wrong persona} & \textbf{policy violation} & \textbf{not safe} & \textbf{etc.} \\
 & & (\%) & (\%) & (\%) & (\%) & (\%) & (\%) \\
\midrule
\textbf{Out-of-Bounds Detection} \\
Generator (IC) + Classifier & 1,471 & 18.10 & 9.31 & 1.61 & 2.49 & 0.07 & 4.66 \\
\hline
\textbf{Response Selection} \\
Retrieve-and-Rerank & 1,230 & 13.17 & 10.68 & 0.72 & 1.53 & 0.00 & 0.24 \\
Retrieve-and-Rerank w/ MC Dropout & 1,272 & 9.82 & 7.58 & 0.36 & 1.66 & 0.00 & 0.22 \\
Retrieve-and-Rerank w/ PPL & 1,300 & \textbf{7.00} & 5.10 & 0.40 & 1.16 & 0.00 & 0.34 \\
\hline
\textbf{Response Generation} \\
Generator (IC) & 985 & 35.83 & 16.05 & 6.24 & 8.66 & 0.17 & 4.68 \\
Generator (MLE) & 1,291 & 4.72 & 3.55 & 0.76 & 0.30 & 0.00 & 0.10 \\
Generator (UL) & 1,497 & \textbf{3.82} & 3.29 & 0.23 & 0.10 & 0.00 & 0.17 \\
\hline
\textbf{Retrieve-fail-Generate} \\
Retrieve-and-Rerank w/ PPL + Generator (UL) & 1,522 & \textbf{2.56} & 2.20 & 0.17 & 0.16 & 0.00 & 0.00 \\
Retrieve-and-Rerank w/ PPL + Generator (UL) + Feedback Data & 1,599 & \textbf{2.00} & 1.88 & 0.00 & 0.10 & 0.00 & 0.00 \\
\bottomrule
\end{tabular}
\end{adjustbox}
\caption{Human evaluation results. As described in Section \ref{sec:feedback}, the crowd workers chat 1:1 with a chatbot as users and correct the inappropriate responses. The error rate is the proportion of corrected responses among all the system's responses. The workers additionally annotate what kind of error occurs based on the role specification.}
\label{tab:model}
\end{table*}




\section{Experiments}
We detail experimental settings and results in this section, including evaluations of the data collected by in-context few-shot learning (Section \ref{sec:dataeval}), comparisons of model variants (Section \ref{sec:model}), and evaluations on system's response qualities (Section \ref{sec:ssa}).

\subsection{Dataset}
\label{sec:dataset}
We built a Korean dialogue dataset for a chatbot system to have casual conversations on a regular basis with senior citizens who live alone.
This dataset was collected using the framework described in Section \ref{sec:data}, assuming a role specification in Table \ref{tab:spec}.
250 dialogue examples with 89 topics (more details are in Appendix \ref{sec:topic}) were used for in-context 1-shot generation. We used 39B size of HyperCLOVA \cite{kim-etal-2021-changes} as generation model (sampling at temperature 0.5 using nucleus sampling \cite{holtzman2019curious} with $P =$ 0.8). %
Table \ref{tab:stat} shows the statistics of the dataset (additional analysis in Appendix \ref{sec:diversity}).
We use 5\% of each for validation sets.



\subsection{Evaluation on Generated Dialogues}
\label{sec:dataeval}

We first assess the quality of the generated dialogues to verify the dialogue generating method described in Section \ref{sec:dialog}.
Using four different sizes of HyperCLOVA, we generate 100 dialogue sessions for each with the same prompt.
We ask the crowd workers to rate on a scale of 1 to 5 whether the generated dialogue satisfies several conditions expected to be controlled through in-context learning (the detailed description of the evaluation criteria is provided in Appendix \ref{sec:humaneval}).
The results are shown in Table \ref{tab:dialog}.
It shows that the larger the model size, the better to meet the conditions by in-context learning, which is also shown in previous studies \cite{NEURIPS2020_1457c0d6, kim-etal-2021-changes}.
In addition, Distinct-1/2 \cite{li-etal-2016-diversity} indicates that the text generated by large models is more diverse.



\subsection{Model Comparison}
\label{sec:model}

\paragraph{Out-of-Bounds Detection}
Table \ref{tab:cls} shows the classification accuracy and F1 score of the trained classifier.
We use generator controlled by in-context learning (IC) as a response prediction model to evaluate the effect of the classifier alone.
For in-context learning, we use the same prompt used to generate the dataset, but the model only generates system's utterances in its turns.
The classifier significantly lowers the error rate of in-context learning (Table \ref{tab:model}), showing the effectiveness of the classifier.
On the other hand, the error rate is relatively higher than those of the best models of response selection and generation.
This is because the classifier is not perfect (about 92\% in accuracy), and even when it properly detects out-of-bounds, the pre-defined questions as alternatives are occasionally incoherent with the contexts.


\begin{table}[t!]
\centering
\begin{adjustbox}{max width=0.85\columnwidth}
\begin{tabular}{ccc}
\toprule
\small \textbf{Training Data (\%)} & \small \textbf{Mean Accuracy\% (std)} & \small \textbf{Mean F1\% (std)} \\
\midrule
\small 10 & \small 87.31 (0.0164) & \small 88.44 (0.0163) \\
\small 20 & \small 89.73 (0.0061) & \small 90.47 (0.0055) \\
\small 100 & \small 91.99 (0.0022) & \small 92.55 (0.0019) \\
\bottomrule
\end{tabular}
\end{adjustbox}
\caption{Classifier results, reporting accuracy and F1 on test set. It shows performance in relation to the amount of training data used.}
\label{tab:cls}
\end{table}

\begin{table}
\centering
\begin{adjustbox}{max width=\columnwidth}
\begin{tabular}{llrrr}
\toprule
\textbf{Model} & \textbf{data} & \textbf{\# of examples} & \textbf{Hits@1/20} & \textbf{Hits@1/100} \\
\midrule
\multirow{2}{*}{Retriever} & Filtered & 47,091 & 93.14 & 83.80 \\
                           & Unfiltered & 227,638 & \textbf{95.27} & \textbf{86.99} \\
\hline
\multirow{2}{*}{Reranker} & Filtered & 47,091 & 97.16 & 90.89 \\
                          & Unfiltered & 227,638 & \textbf{97.55} & \textbf{91.70} \\
\bottomrule
\end{tabular}
\end{adjustbox}
\caption{Hits@1/$K$ of retriever and reranker on the validation set. Hits@1/$K$ measures recall@1 when ranking the gold label among a set of $K-1$ other random candidates.}
\label{tab:ret}
\end{table}


\begin{table*}[hbt!]
\centering
\begin{adjustbox}{max width=\textwidth}
\begin{tabular}{lrrrr}
\toprule
& \multicolumn{2}{c}{\textbf{Response Selection}} & \multicolumn{2}{c}{\textbf{Response Generation}} \\
\cmidrule(lr){2-3}\cmidrule(lr){4-5}
 & \textbf{proportion} & \textbf{error rate} & \textbf{proportion} & \textbf{error rate} \\
\textbf{Model} & (\%) & (\%) & (\%) & (\%) \\
\midrule
Retrieve-and-Rerank w/ PPL + Generator (UL) & 68.20 & 2.50 & 31.80 & 2.68 \\
Retrieve-and-Rerank w/ PPL + Generator (UL) + Feedback Data & 63.70 & 2.12 & 36.30 & 1.77 \\
\bottomrule
\end{tabular}
\end{adjustbox}
\caption{Evaluation results of each component in the Retrieve-fail-Generate pipeline. It shows the proportion and error rate of returned responses from response selection and generation models.}
\label{tab:feedback}
\end{table*}

\begin{table}
\centering
\begin{adjustbox}{max width=\columnwidth}
\begin{tabular}{lrr}
\toprule
\textbf{Method} & \textbf{positive} & \textbf{negative} \\
\midrule
In-context Learning & 2.65 & 2.74 \\
Likelihood Training & 2.07 & 2.47 \\
Unlikelihood Training & 2.48 & 46.70 \\
\bottomrule
\end{tabular}
\end{adjustbox}
\caption{Perplexity (PPL) of generative models on validation set of filtered data.}
\label{tab:ppl}
\end{table}

\paragraph{Response Selection}
We fine-tune the response selection models on positive examples of the filtered data and automatically evaluate them by measuring Hits@1/$K$ \cite{roller-etal-2021-recipes} on the validation set.
Results are shown in Table \ref{tab:ret}. 
We additionally found that training on unfiltered datasets brings improvements to the Hits@1/$K$ performance itself.
Therefore, we use the models that trained on unfiltered dataset in the subsequent experiments.
Response candidates are limited to system responses within positive examples (unique system's turns of filtered data in Table \ref{tab:stat}).
And we also validate the proposed methods for predicting unanswerable contexts, and determine the thresholds for each (further details are given in Appendix \ref{sec:valid}).




Table \ref{tab:model} shows the error rate of the response selection models.
The model that does not predict unanswerable contexts (Retrieve-and-Rerank) has a higher error rate in `not sensible' than others.
The case of using PPL as the method for predicting unanswerable contexts shows a lower overall error rate than the case of using MC Dropout, and the proportions of the total contexts predicted as unanswerable are similar at 4.23\% and 3.85\% for PPL and MC Dropout, respectively.
The results also show the error types from the models.
Although only the filtered utterances are used as response candidates, `wrong persona' and `policy violation' appear in responses.
It seems that a few unfiltered utterances remain in the response candidates, since the human filtering is not perfect. 
Or even the same utterance can cause errors depending on the context.
For example, it is possible to agree with when a user calls the system by a different name.



\paragraph{Response Generation}

We compare three ways to train generators; in-context learning (IC), likelihood training (MLE), and unlikelihood training (UL).
We measure the perplexity of the three models on positive and negative examples and Table \ref{tab:ppl} shows the results.
The difference between the PPL of the positive examples and the negative examples is the smallest in in-context learning.
When trained on positive examples with likelihood training, the difference increases slightly, because the PPL of the positive examples is lowered.
When adding unlikelihood training, the PPL for negative examples increase significantly, \footnote{\citet{li-etal-2020-dont} has also found a large gap in PPL scores between positives and negatives.} which mean the model is less likely to generate out-of-bounds utterances.

Table \ref{tab:model} shows the error rate of each model.
Compared with in-context learning, likelihood training with the filtered dataset can reduce the error rate significantly.
Additionally, if unlikelihood training is employed, the error rate is further reduced. A similar trend can be found in all types of errors.


\paragraph{Retrieve-fail-Generate}

We also experiment with a Retrieve-fail-Generate model consisting of the best configurations for response selection (PPL) and generation (UL) models.
Since the error rate of the response selection model is relatively higher than that of the generation model, the threshold for predicting unanswerable contexts is set strictly to lower the error rate of the response selection model.
Table \ref{tab:feedback} shows the error rates of responses returned from response selection and generation models, respectively.
The results indicate that both error rates are lower when the models are included in a pipeline than when they are used separately, and the overall error rate decreases accordingly.
The response selection model returns the responses within the candidates extracted from the positive examples of the trainset,
so that the flow of the conversation is not dispersed and tends to be similar to the trainset.
As a result, the Retrieve-fail-Generate model shows the lowest error rate among all models (Table \ref{tab:model}).

\paragraph{Feedback Pipeline}

The best model is further trained on the human-bot dialogues collected during the model evaluation process, as depicted in Section \ref{sec:feedback}.
Both response selection and generation models are newly initialized and trained.
As a result, all types of error rates are consistently reduced (Table \ref{tab:model}), and the error rates of both the response selection and generation models are decreased (Table \ref{tab:feedback}).
The effect is stronger on the response generation.


\begin{table}
\centering
\begin{adjustbox}{max width=\columnwidth}
\begin{tabular}{lrrr}
\toprule
\textbf{Method} & \textbf{Sensibleness} & \textbf{Specificity} & \textbf{SSA} \\
\midrule
Human & 95.48 & 82.96 & 89.22 \\
Retrieve-fail-Generate + Feedback Data & 94.00 & 77.50 & 85.75 \\
\bottomrule
\end{tabular}
\end{adjustbox}
\caption{Interactive SSA results.}
\label{tab:ssa}
\end{table}


\subsection{Response Quality} %
\label{sec:ssa}
To assess the overall response quality of the proposed chatbot system, we use the sensibleness and specificity average (SSA) metric \cite{adiwardana2020towards}, which is shown to have a strong correlation with asking raters how humanlike the model is.
This metric is a average of two scores: sensibleness and specificity. Sensibleness measures whether a model's responses make sense in context and do not contradict anything that was said earlier, while specificity measures whether a response is specific to a given context.
However, exact comparison with the scores in \citet{adiwardana2020towards} is difficult, because of the static role of our chatbot system and language discrepency in phrasing of questions.
Therefore, We re-estimate human interactive SSA in our experiments. 
To collect human-human conversations, we transcribe 100 call speeches between users and workers who play system's role.
And we collect 100 human-bot conversations by allowing the crowd workers to chat with the system.
Labeling was conducted by independent crowd workers with majority voting of 5 workers per turn.

The results are given in Table \ref{tab:ssa}. 
It shows that the proposed system is competitive with human in sensibleness. And the majority of the responses from the system are labeled as specific, which allows us to conclude that the proposed system achieves low error rate with non-generic responses.
We also report agreement and Krippendorff's alpha \cite{krippendorff2011computing} for measure of consistency of crowd workers in Appendix \ref{sec:agree}.

\section{Discussion}

Although our methods achieve the low error rates in human interactive evaluations,
the results have some limitations. The results should be regarded as the error rates of typical conversations without adversarial attack. Because the annotators are instructed to participate in the chat as if they were typical users, they did not try as many conversations that could induce toxic words from the model. This may be the reason why the toxicity is close to zero as shown in Table \ref{tab:model}.



The human filtering process in the proposed data collection framework has room to be more efficient. Since the accuracy of the classifier is comparable even when just 10\% of the total data is used (Table \ref{tab:cls}), it is expected that the filtering cost can be reduced by adding a model filtering process before human filtering, which is similar to the method proposed in \citet{sun-etal-2021-adding}.



\section{Conclusion}
We present a framework for building role specified open-domain dialogue systems from scratch. We propose leveraging large-scale LMs to generate supervisory datasets for training dialogue systems with arbitrary roles with minimal effort for manually composing dialogues. Our research also analyzes several model architectures for the task.
By extensive experiments, we demonstrate the effectiveness of the collected data and modeling approaches in terms of satisfying role constraints and improving dialogue abilities.
We argue that our framework can be extended to implement dialogue systems with various roles and characters, even when available datasets are few.


\section{Ethical Considerations}
Workers annotating the dataset we built were hired on a part-time basis and compensated based on the number of working hours.
They were compensated with 9,000 won per hour, which was somewhat higher than the Korean minimum wage at the time they worked.
Appropriate instructions for the use of collected data were given at the time of contract and consent was obtained.
We will release our dataset in CC-BY-NC-SA license.\footnote{\url{https://creativecommons.org/licenses/by-nc-sa/2.0/}}

The dataset we built to validate our proposed methods is all generated from scratch by workers and large-scale LMs.
Although there is no user data in the dataset, pre-trained language models are known to exhibit private details in their outputs \cite{DBLP:journals/corr/abs-2012-07805}, as well as social biases \cite{bender2021dangers, bordia-bowman-2019-identifying, garrido2021survey, shwartz-choi-2020-neural} and toxic contents \cite{gehman-etal-2020-realtoxicityprompts}.
To address these concerns, we determined categories and criteria for harmful texts based on legal and ethical considerations provided by experts in our group, and we instructed annotators to filter the dataset using these criteria.
However, due to missing annotations and cultural or social biases, this may be imperfect. To mitigate this, we had multiple crowd workers annotate the same data. In addition, because the users in the dataset are regarded to be a vulnerable population, our group's ethical consultation looked through the issues that would be sensitive to them, and dialogues containing these topics were also eliminated.

Despite these efforts, using this dataset to directly train end-to-end chatbot models can involve certain risks, 
due to the lack of controllability and interpretability in end-to-end neural response prediction models.
And it should not be overlooked that they may cause some potential harm, 
even though the chatbot systems can help reduce social loneliness of the user population.
For example, a user can become emotionally attached to a bot, even codependent on it, which can divert attention away from real-world relationships and cause distress if the chatbot fails.
It's also worth noting that a chatbot can be programmed to impersonate a real person and be used for phishing and fraud.
During such conversations, users may provide private and sensitive information, such as specific health conditions and private attributes, which could be exploited if it falls into the wrong hands. 
For this reason, when incorporating this dataset in real-world applications, the application developers should ensure that it is used safely and ethically.


Since our proposed framework also can be used for building another dataset and chatbot system with arbitrary specifications, it is not exempt from the possibility of propagating linguistic biases and toxicity.
Similar to \citet{xu-etal-2021-bot}, we are in progress continuously reducing the unsafe texts from LM itself through our feedback pipeline and unlikelihood training, which might be included in our future works.


\section*{Acknowledgements}
The authors thank all the members of CLOVA and AI Lab of NAVER for devoted supporting and discussion.
In particular, they would like to thank Gichang Lee, Hwijeen Ahn, and the members of CLOVA Conversation for their dedicated efforts to facilitate the large-scale models.
In addition, the authors thank Hyeri Kim, Hyunjung Park, and Yuin Jeong for their great help in designing the role of the chatbot.
Also, they thank Jeeseung Han for technically supporting the serving and testing environments.
Finally, the authors thank Nako Sung for his support and valuable comments on this project.

