Fine Grained Knowledge Transfer for
Personalized Task-oriented Dialogue Systems
Kaixiang Mo† and Yu Zhang† and Qiang Yang† and Pascale Fung‡

arXiv:1711.04079v1 [cs.CL] 11 Nov 2017

Hong Kong University of Science and Technology, Hong Kong, China
†
{kxmo, zhangyu, qyang}@cse.ust.hk, ‡ pascale@ece.ust.hk

Abstract
Training a personalized dialogue system requires a lot of
data, and the data collected for a single user is usually insufficient. One common practice for this problem is to share
training dialogues between different users and train multiple
sequence-to-sequence dialogue models together with transfer learning. However, current sequence-to-sequence transfer
learning models operate on the entire sentence, which might
cause negative transfer if different personal information from
different users is mixed up. We propose a personalized decoder model to transfer finer granularity phrase-level knowledge between different users while keeping personal preferences of each user intact. A novel personal control gate is
introduced, enabling the personalized decoder to switch between generating personalized phrases and shared phrases.
The proposed personalized decoder model can be easily combined with various deep models and can be trained with reinforcement learning. Real-world experimental results demonstrate that the phrase-level personalized decoder improves the
BLEU over multiple sentence-level transfer baseline models
by as much as 7.5%.

Introduction
Task-oriented dialogue systems aim to help a user to finish
a certain task with dialogues, and it can be categorized into
rule-based systems and learning-based systems. Learningbased dialogue systems do not require the predefined dialogue state, which are more general and suitable for the
situation when exact dialogue states are hard to define. In
particular, neural network based dialogue systems do not require the predefined templates and are more flexible. In this
paper, we focus on neural network based task-oriented dialogue systems.
Personalized dialogue systems can help the target user to
complete a task faster and make the dialogue more efficient.
In a personalized dialogue system, the personal information
and preference of the target user are recorded and utilized,
thus the personal dialogue policy can generate personalized
sentences and speed up the dialogue process for the target
user. In personalized dialogue systems, the dialogue policy
and response generation for each user is different. Training
a whole personalized dialogue system requires a lot of data
Copyright c 2018, All rights reserved.

ࢎ௨ǡ௘
ೠǡೣ
ே೙
hƐĞƌ
ĐƚŝŽŶ
sĞĐƚŽƌ

Ϯ͗ŝĂůŽŐƵĞ
^ƚĂƚĞ
dƌĂĐŬŝŶŐ
;^dͿ

ϭ͗^ƉŽŬĞŶ
>ĂŶŐƵĂŐĞ
hŶĚĞƌƐƚĂŶĚŝŶŐ
;^>hͿ

ࢄ௨௡
hƐĞƌdĞǆƚ
hƚƚĞƌĂŶĐĞ

ࢎ௨ǡ௖
௡

ŝĂůŽŐ
^ƚĂƚĞ
sĞĐƚŽƌ

ZĞǁĂƌĚ
‫ݎ‬௡௨

ϯ͗ŝĂůŽŐƵĞ
WŽůŝĐǇ
>ĞĂƌŶŝŶŐ
;W>Ϳ

ࢎ௨ǡௗ
௡ǡ଴
^ǇƐƚĞŵ
ĐƚŝŽŶ
sĞĐƚŽƌ

ϰ͗WĞƌƐŽŶĂůŝǌĞĚ EĂƚƵƌĂů>ĂŶŐƵĂŐĞ
'ĞŶĞƌĂƚŝŽŶ ;E>'Ϳ
^ŚĂƌĞĚ
'ĞŶĞƌĂƚŽƌ

WĞƌƐŽŶĂů
'ĞŶĞƌĂƚŽƌ

ࢅ௨௡
WĞƌƐŽŶĂůŝǌĞĚ
^ǇƐƚĞŵdĞǆƚ
ZĞƐƉŽŶƐĞ

Figure 1: Framework for Personalized Task-oriented Dialogue System
and the data collected from a single user is usually insufficient. Multi-tasking can be used to transfer dialogue knowledge across different users by sharing training dialogues.
However, current neural network transfer models work
on the entire sentence, but the phrases concerning personal
information from different users should not be transferred
across users at all. For example, if the dialogue data from a
sugar lover is used to train a personalized dialogue system
for a diabetes patient, the system might recommend sweet
drinks to the patient and cause a disaster. Hence, we think
that knowledge transfer in personalized dialogue systems
should be conducted in finer granularity.
In this paper, we propose a personalized decoder that can
transfer shared phrase-level knowledge between different
users while keeping the personalized information of each
user intact. A novel personal control gate is introduced in
the proposed personalized decoder, enabling the decoder
to switch between generating shared phrases and personal
phrases. For example in Figure 2(b), “Hot Latte” is a personal phrase and “Still” and “?” are shared phrases. The personalized decoder can generate different personal phrases
for different users in a sentence, while the knowledge for
the shared phrases are shared among all users. The proposed
personalized decoder can be easily used with various base
models to achieve fine-grained phrase-level dialogue knowledge transfer between different users. Real-world experimental results show that the personalized decoder can im-

prove the BLEU score over multiple sentence-level transfer
models by as much as 7.5%.

Related work
Learning based task-oriented dialogue systems (Casanueva
et al. 2015; Genevay and Laroche 2016; Gašic et al. 2013;
Gasic et al. 2014; Williams and Zweig 2016; Bordes and
Weston 2016) can select appropriate answers from a set
of predefined answers. Gašic et al. (Gašic et al. 2013;
Gasic et al. 2014) used transfer learning in the dialogue
management module to extend a dialogue system to handle previous unseen concepts. Williams et al. (Williams and
Zweig 2016; Bordes and Weston 2016) used learning methods in spoken language understanding and dialogue management. These methods require the predefined answer candidates or templates. Wen (Wen et al. 2015; Wen et al. 2016a;
Wen et al. 2016b) proposed to use trainable modules for each
part of the dialogue system which does not require predefined answer candidates or templates. However, this model
still requires the predefined slots for dialogue policy learning, which limits its application.
Neural based dialogue response generation systems require neither the answer candidates/templates, nor the predefined slots for dialogue policy. Sequence to sequence
(seq2seq) models and their variants (Bahdanau, Cho, and
Bengio 2014; Sutskever, Vinyals, and Le 2014; Sordoni et
al. 2015; Serban et al. 2015; Li et al. 2016) are widely used
to model dialogues. However, data collected from each user
is insufficient for training a personalized dialogue system.
Multi-task learning is used to transfer knowledge in sequence to sequence model. Luong et al. (Luong et al. 2015)
proposed to share encoder/decoder across different tasks in
order to achieve knowledge transfer. However, these knowledge transfer works on the granularity of sentences. Since
different persons have different preferences and require different dialogue policies and responses, directly transferring
sentences across different users might lead to the negative
transfer.
In this paper, a personalized decoder is proposed for response generation, which is capable of transferring dialogue
knowledge, and it can easily be combined with many models including seq2seq (Sutskever, Vinyals, and Le 2014) and
HRED (Serban et al. 2015).

Problem
In this section, we first define notations and then present the
problem settings.

Notation
In this paper, matrices are denoted in a bold capital case,
column vectors are in a bold lower case, and scalars are in a
lower case. Questions are denoted by X = {Xun }N
n=1 , where
N is the number of turns in a dialogue, the superscript u deu,x
Nn
notes users, Xun = {xun,t }t=1
denotes the n-th question,
and Nnu,x is the number of words in Xun . Responses are denoted by Y = {Ynu }N
and the n-th response Ynu is den=1u,y
Nn
u
u
noted by Yn = {yn,t }t=1 , where Nnu,y is the number of

ࢎ௨ǡௗ
௡ǡ଴

ࢎ௨ǡௗ
௡ǡଵ

ࢎ௨ǡௗ
௡ǡଶ

ࢎ௨ǡௗ
௡ǡଷ

^ƚŝůů

,Žƚ

>ĂƚƚĞ

ࢎ௨ǡௗ
௡ǡସ
͍

(a) Response generation with a basic decoder. hu,c
is not
n
shown since it is the same for all time step t.
ࢎ௨ǡௗ
௡ǡ଴

ࢎ௡ǡଵ

௨ǡ௚

ࢎ௡ǡଶ

௨ǡ௚

ࢎ௡ǡଷ

௨ǡ௚

ࢎ௡ǡସ

ࢎ௨௡ǡଵ

ࢎ௨௡ǡଶ

ࢎ௨௡ǡଷ

ࢎ௨௡ǡସ

ࢎ௨ǡௗ
௡ǡଵ

ࢎ௨ǡௗ
௡ǡଶ

ࢎ௨ǡௗ
௡ǡଷ

ࢎ௨ǡௗ
௡ǡସ

Ϭ

^ƚŝůů

ϭ

,Žƚ

ϭ

>ĂƚƚĞ

௨ǡ௚

Ϭ

Ϭ

͍

(b) Personalized response generation with the proposed personalized decoder. The personal control gate at different time
steps are denoted in gray circles.

Figure 2: Response generation comparison
words in Ynu . To be consistent, a dialogue turn is indexed by
n and a word is indexed by t.

Problem Definition
Given the conversation history of multiple users, we aim to
learn an end-to-end personalized task-oriented dialogue system for each user. The input of the problem are:
1. Historical dialogue sessions T u = {Xun , Ynu , rnu }n=1 of
each user, where rnu is the reward obtained at n-th dialogue turn.
2. Personal word label Onu = {oun,t }N
t=1 for each word in
Ynu , where oun,t = 1 means that xun,t is a personal word
and oun,t = 0 means the word xun,t is a general word.
The output of the problem are:
1. A dialogue policy π u for each user u, which generates a response Ynu for each dialogue history Hnu =
u
{{Xui , Yiu }n−1
i=1 , Xn }.

Personalized Decoder
In this section, we firstly introduce the proposed personalized decoder, then introduce the combination of the personalized decoder with two models, and finally present the
training method.

Decoder for Response Generation
In this section, we introduce the mathematical formulation
of the proposed personalized decoder.

Basic Decoder The hidden state for the t-th word in the
n-th turn is defined as
d u,d
d u
hu,d
+ Vd hu,c
n,t = tanh(W hn,t−1 + U eŷn,t−1
n ),
u
u
where eŷn,t−1
is the word embedding of the last word ŷn,t−1
in the same sentence, and tanh(·) denotes the hyperbolic
tangent function. The decoder RNN in Figure 2(a) takes hu,d
n,0
1
and hu,c
as
inputs
and
then
generates
the
response
word
by
n
word, where hu,c
n is the embedding vector used to generate
the current response. The probability of generating the next
u
u
u
word ŷn,t
= yn,t
given hu,d
n,t and ŷn,t−1 is
u
u,d
u
ω(hu,d
n,t , ŷn,t−1 ) = H0 hn,t + E0 eŷn,t−1 + bo

(1)

u
u
u
T
u,d
u ω(h
g(hu,d
n,t , ŷn,t−1 )
n,t , ŷn,t−1 , yn,t ) = oyn,t

(2)

u
u
p(ŷn,t
= yn,t
)=

u
u
exp(g(hu,d
n,t , ŷn,t−1 , yn,t ))
P
u,d
u
0
∀y 0 exp(g(hn,t , ŷn,t−1 , y ))

(3)

where ov is the output embedding for word v, and H0 , E0
and bo are parameters.
Personalized Dialogue Decoder for Phrase-level Transfer Learning In this section, we present the proposed personalized decoder as illustrated in Figure 2(b). While the
sentence-level transfer is to transfer entire sentences, the
proposed personalized decoder is on the phrase level and is
to transfer a shared fraction of the sentences to the target domain, where a phrase is a short sequence of words containing a coherent meaning, for example, an address. In order to
achieve maximum knowledge transfer and to avoid negative
transfer caused by differences in user preferences, the proposed personalized decoder has a shared component and a
personalized component. In order to learn to switch between
the shared and personal components in the phrase level, we
introduce a personal control gate oun,t for each word, which
is learned from the training data.
Given the the embedding vector for the n-th response hu,c
n
u
and initial hidden state hu,d
for
word
ŷ
,
the
initial
states
n,0
n,0
are computed as
u,d
u,d u
u
hu,g
n,0 = hn,0 , hn,0 = hn,0 , ôn,0 = 0
u,g = 0
u
eŷn,0
= 0, eŷn,0

hu,g
n,t

where
is the hidden state for the shared component, and
u,g
ŷn,t
records the last word generated by the shared component, hun,t is the hidden state for the personal component,
u
and hu,d
n,t is the hidden state for generating the word ŷn,t .
The shared component adopts the GRU model to capture
the long-term dependency and is shared by all users. Specifically, at each time step t, the shared component is defined
as
g
u,g
zun,t = σ(Wzg hu,g
n,t−1 + Uz eŷ

+ Vzg hu,c
n + bz )

g
u,g
run,t = σ(Wrg hu,g
n,t−1 + Ur eŷ

+ Vrg hu,c
n + br )

n,t−1

n,t−1

g
u
h̃u,g
n,t = σ(Wh (rn,t
u
ĥu,g
n,t = zn,t

g
u,g
hu,g
n,t−1 ) + Uh eŷ

n,t−1

u
hu,g
n,t−1 + (1 − zn,t )

+ Vhg hu,c
n + bh )
h̃u,g
n,t ,

1 u,c
hn is not shown in Figure 2(a) since it is the same for all time
step t.

where denotes the element-wise product between vectors
or matrices, σ(·) is the sigmoid function, zun,t is the update
gate, run,t is the forget gate, and ĥu,g
n,t is the tentative updated
hidden state. If the t-th word is a shared word (i.e., ôun,t = 0),
then we update the shared hidden state and last general word
u,g remain unchanged.
as usual and otherwise hu,g
n,t and eŷt
u,g can be updated as
Thus hu,g
and
e
ŷt
n,t
u
hu,g
n,t = (1 − ôn,t )

u
ĥu,g
n,t + ôn,t

eŷtu,g = (1 − ôun,t )

u
eŷt−1
+ ôun,t

hu,g
n,t−1
u,g .
eŷt−1

The personal component is a RNN model, which generates personalized sequence based on sentence context hu,g
n,t
from the shared component. There is an separate RNN
model for each user. At each time step t, the personal comu
ponent receives eŷt−1
, ôun,t , hun,t−1 and hu,g
n,t−1 as inputs and
outputs ĥun,t , which is defined as
u
ĥun,t = σ(Wu hun,t−1 + Uu eŷn,t−1
+ Vu hu,g
n,t−1 ).

The personal hidden state will be update as
hun,t = (1 − ôun,t )

u
hu,g
n,t + ôn,t

ĥun,t .

hun,t equals ĥun,t if the control gate is on corresponding to
ôun,t = 1. If ôun,t equals 0, hun,t will take the value of hu,g
n,t .
The personal control gate oun,t is binary, i.e., oun,t ∈ {0, 1}.
The predicted control gate ôun,t at time t is a function of
u
2
u
ôun,t−1 , hu,g
n,t−1 , hn,t−1 , and eŷn,t−1 as
u

p(ôn,t = 1) =


g u,g
 σ(Wo
hn,t−1 + Ug
o eŷ u


u u
σ(Wo
hn,t−1

+

n,t−1
Uu
u
o eŷn,t−1

+ bo )

if ôu
n,t−1 = 0

bu
o)

if ôu
n,t−1 = 1

+

.
(4)

ôun,t decides whether to use the personal component to generate the next word. hu,d
n,t is defined as
u
hu,d
n,t = (1 − ôn,t )

u
hu,g
n,t + ôn,t

hun,t ,

(5)

where hu,d
n,t is the hidden vector that directly generates the
u
and the probability of generating the next
next word ŷn,t
u
is defined by the generation process in Eqs. (1-3).
word yn,t
The decoding procedure is as follows:
u,d
u
u
u,g
u
1. Initialize hu,g
n,0 , hn,0 , ôn,0 , eŷn,0 , eŷn,0 based on hn,0 and
u
u,c
u
hn . ôn,0 is initialized to be 0 and eŷn,0
is initialized to
be a zero vector.
u,g
u
u
2. Compute ôun,t based on hu,c
n , ôn,t−1 , hn,t−1 , hn,t−1 and
u
eŷt−1
with Eqs. (4).
u,d
u
u
3. Compute hu,g
n,t , hn,t and hn,t based on ôn,t .
u
4. Generate ŷn,t
based on hu,d
n,t with Eqs. (5).
5. Repeat step 2 to step 4 until the ending symbol.
The shared and personal components can be trained together
with reinforcement learning as illustrated in the parameter
learning section.
Compared with the basic decoder, the personalized decoder is novel in the following aspects:
2
In training process, the ground truth oun,t is used as label to
train the prediction function for ôun,t .

,ŝŐŚ>ĞǀĞůZEEĨŽƌŝĂůŽŐƵĞ^ƚĂƚĞ
dƌĂĐŬŝŶŐ
^ƚĂƚĞϮ
^ƚĂƚĞϭ
dƌĂŶƐŝƚŝŽŶ

ŝĂůŽŐƵĞ
WŽůŝĐǇ
>Žǁ>ĞǀĞůZEEĨŽƌ
^ƉŽŬĞŶ>ĂŶŐƵĂŐĞ
hŶĚĞƌƐƚĂŶĚŝŶŐ

/ŶƉƵƚܺ௡௨ ͗/ǁŽƵůĚůŝŬĞĂĐƵƉŽĨĐŽĨĨĞĞ͘

ࢎ௨ǡௗ
௡ǡ଴

෢௡௨ ͗
WƌĞĚŝĐƚĞĚܱ
WĞƌƐŽŶĂůŝǌĞĚ
ĞĐŽĚĞƌ
௨
WƌĞĚŝĐƚĞĚܻ෢
௡ ͗

ࢎ௡ǡଵ

௨ǡ௚

ࢎ௡ǡଶ

௨ǡ௚

ࢎ௡ǡଷ

௨ǡ௚

ࢎ௡ǡସ

ࢎ௨௡ǡଵ

ࢎ௨௡ǡଶ

ࢎ௨௡ǡଷ

ࢎ௨௡ǡସ

ࢎ௨ǡௗ
௡ǡଵ

ࢎ௨ǡௗ
௡ǡଶ

ࢎ௨ǡௗ
௡ǡଷ

ࢎ௨ǡௗ
௡ǡସ

^ƚŝůů

,Žƚ

>ĂƚƚĞ

Ϭ

ϭ

௨ǡ௚

ϭ

Ϭ

͍

dƌƵĞKƵƚƉƵƚܻ௡௨

͗^ƚŝůůŚŽƚůĂƚƚĞ͍
dƌƵĞWĞƌƐŽŶĂůtŽƌĚ>ĂďĞůܱ௡௨ ͗Ϭϭ ϭ Ϭ

Figure 3: Model framework for personalized HRED, obtained by combining HRED with Personalized Decoder
1. Knowledge is transferred in the phrase level rather than
the sentence level.
2. There is a shared component for knowledge transfer and
a personalized component for each user in order to avoid
negative transfer caused by different data distributions.
3. The personal control gate can learn when to use the shared
component and when to use the personal component to
generate a fluent and personalized response.

Combining HRED with Personalized Decoder
In this section, we show how the proposed personalized decoder can be combined with hierarchical recurrent encoderdecoder (HRED) (Serban et al. 2015) to make a personalized
HRED, whose model is shown in Fig. 3.
In the personalized HRED, there are a low-level encoding RNN, a high-level RNN, a dialogue policy and a personalized decoder. The low-level encoding RNN for spoken
language understanding encodes user utterances into a user
action vector, the high-level RNN is responsible for dialogue
state tracking, the dialogue policy maps the dialogue state to
the action vector, and the personalized decoder is responsible for generating words as the response.
Spoken Language Understanding with Word Encoder
The low-level encoder RNN for spoken language understanding maps each Xun = {xun,1 , xun,2 , · · · , xun,Nnu,x } to a
fixed dimension vector hu,e
u,x as
Nn
e u,e
e
hu,e
),
n,t = tanh(W hn,t−1 + U exu
n,t

where Ue is the input embedding matrix and We is the
weight matrix corresponding to hidden state (word-level)
transition function. Similarly we can define a mapping from
u
u
u
the response Ynu = {yn,1
, yn,2
, · · · , yn,N
u,y } to the ren
u,e
sponse vector h̄Nnu,x as
e u,e
e u
h̄u,e
).
n,t = tanh(W h̄n,t−1 + U eyn,t

Dialogue State Tracking with Sentence RNN The highlevel sentence RNN tracks dialogue states based on all previous sentences in the dialogue. This RNN takes h̄u,e
,
n−1,N u,y
n−1

u,c
hu,e
u,x and hn−1 as the input to generate next dialogue
n,Nn

state hu,c
n as:
c u,c
c u,e
h̄u,c
n−1 = tanh(W hn−1 + U h̄n−1,N u,y )
n−1

c u,c
c u,e
hu,c
u,x ),
n = tanh(W h̄n + U hn,Nn
u,e
where h̄u,e
u,y and h
u,x are encoding vectors of senn−1,Nn−1
n,Nn
c
u
u
tences Yn−1 and Xn , U is the embedding matrix for the
input, and Wc is the weight matrix corresponding to the hidden state (sentence-level) transition function.

Dialogue Policy with Linear Transformation Given the
u,d
current dialogue state hu,c
n , the action vector hn,0 is calculated as
u,c
hu,d
n,0 = tanh(D0 hn + b0 )
where D0 and b0 are the policy parameters.
Response Generation with Personalized Decoder Then
u,d
the personalized decoder takes hu,c
n and hn,0 as inputs to
generate the response word by word.
The joint probability of (Hnu , Onu , Ynu ) is given by:
p(Hnu , Onu , Ynu ) = p(Hnu )p(Onu , Ynu |Hnu )
p(Hnu ) =

n−1
Y

p(Oku , Yku |Hku ),

k
N u,y

n
is the collection of control gate
where Onu = {oun,t }t=0
variables for sentence Ynu .

Combining Seq2Seq Model with Personalized
Decoder
The proposed personalized decoder can be combined with
the popular seq2seq model proposed in (Sutskever, Vinyals,
and Le 2014). We can build a personalized seq2seq model by
replacing the original decoder in the seq2seq model with the
proposed personalized decoder. The whole encoder and the
common component of the personalized decoder are shared
across all users, while each user has his own personal component in the personalized decoder.
Moreover, the proposed personalized decoder can also be
easily combined with many other models to achieve knowledge transfer.

Parameter Learning
The whole model is trained in an end-to-end manner with
reinforcement learning (Sutton and Barto 1998) such that the
model will generate personalized system response according
to current dialogue state to maximize the future cumulative
reward.
Reward The agent will receive general rewards and personal rewards, and the total reward is the sum of general reward and personal reward. The general and personal rewards
will be received under the following conditions:
1. Personal rewards of 0.3 will be received when the user
confirms the suggestion of the agent. This is related to the
personal information of the user. For example, the user
could confirm the address suggested by the agent.
2. General rewards of 0.1 will be received when the user provides the information about the target task.
3. General rewards of 1.0 will be received when the system
helps the user finish the target task successfully.
4. General reward of −0.2 will be received by the agent
when the user rejects to proceed if the system is generating non-logical responses. A reward of −0.05 will be
received for each the dialogue turn.
Loss function We use the policy gradient method to learn
the parameter of the model. Specifically, we use the REINFORCE (Williams 1992) algorithm. The training set consists of a set of trajectories T u = {Hnu , Onu , Ynu , rnu }, which
records actions provided by the user. We denote all these parameters by Θ, and the model policy is denoted by π.u
We define the return of the trajectory J T
=
PN
n−1 u
u
γ
r
where
r
is
the
reward
obtained
at
n-th
dian
n
n=1
logue turn. Then the loss function is defined as the expected
reward of the policy under all trajectory T u :
Z
J(Θ) =
p(T u |Θ)dT u ,

none-transfer versions. Note that we do not assume the predefined dialogue states or templates, thus rule-based systems
do not apply here. All methods are listed as follows:
1. None-transfer seq2seq (Sutskever, Vinyals, and Le 2014)
(denoted by “S2S”): The sequence to sequence model is
trained only with data from each individual user without
transfer learning.
2. Sentence-level transfer seq2seq (Luong et al. 2015) (denoted by “ST-S2S”): The sequence to sequence model
trained in a multi-task setting with both the encoder and
the decoder shared across all users. Sentence-level knowledge is transferred.
3. Sentence-level encoder transfer seq2seq (Luong et al.
2015) (denoted by “ST-E-S2S”): The sequence to sequence model is trained in a multi-task setting with the
encoder shared across all users but a different decoder for
each target user. Sentence-level knowledge is transferred.
4. Personalized phrase-level transfer seq2seq (denoted by
“PT-S2S”): The sequence to sequence model is equipped
with the proposed personalized decoder and trained in the
multi-task setting. Phrase-level knowledge is transferred.
5. None-transfer HRED (Serban et al. 2015) (denoted by
“HRED”): The HRED model is trained only with data
from each individual user, without transfer learning.
6. Sentence-level transfer HRED (denoted by “ST-HRED”):
The HRED model is trained in a multi-task setting, while
sentence-level knowledge is transferred.
7. Personalized phrase-level transfer HRED (denoted by
“PT-HRED”): The personalized HRED model with the
proposed personalized decoder is trained in a multi-task
setting. Phrase-level knowledge is transferred.

Table 1: Statistics of the dataset

Tu

whose gradient can be computed as
∆Θ J(Θ) = E{

N
X

u

u
∆Θ log πΘ (Hn
, Onu , Ynu )JnT }

n=0
u
JnT

PN

k−n u
rk
k=n γ

where
=
is the future cumulative reward and N is the total number of dialogue turns.
Optimization We adopt the Adam algorithm (Kingma and
Ba 2014) to optimize our model. We train the model on data
collected from different users, where the shared parameters
are updated in each iteration while the personalized parameters are updated based on data collected from the corresponding user only.

Experiments
In this section, we experimentally verify the effectiveness
of the personalized decoder by conducting experiments on a
real-world dataset and a simulation dataset.
We compare the proposed two personalized phrase-level
transfer models with their sentence-level counterparts and

Dataset
Simulation
Real

Train Set
Users Dialogues
10
50
52
464

Test Set
Users Dialogues
10
2,000
52
831

Simulation Experiments
In this section, we conduct experiments on a simulation
dataset.
We build a rule-based user simulator for multi-turn coffee
ordering services. The user simulator will order his favorite
coffee with probability 0.8 and try new coffee with probability 0.2. The simulated user will answer questions about
the coffee type, the temperature, the cup size and the delivery address. The user will give rewards according to the
reward function defined in the reward section. All models
will select an answer with the highest generative probability from a set of answer candidate templates, which is filled
with the confirmed information obtained from the simulation user. In order to obtain ground truth dialogues, we design a rule-based ground truth agent which will choose the
best reply with probability 0.8 and choose a random reply
with probability 0.2. We have 10 simulated users in total.

1.0
0.9
0.8

5:HRED
6:ST-HRED
7:PT-HRED

0.7

1:S2S
2:ST-S2S
3:ST-E-S2S
4:PT-S2S

0.6
0.5

Slot Error Rate

1:S2S
2:ST-S2S
3:ST-E-S2S
4:PT-S2S

0.4

BLEU

0.7

5:HRED
6:ST-HRED
7:PT-HRED

0.3

0.6

0.2

0.5

0.1

0.4

1

2

3

4

5

6

7

0.0

Figure 4: BLEU for the simulation dataset

2

3

4

5

6

7

Figure 5: Slot-error-rate for the simulation dataset
2.5

1:S2S
2:ST-S2S
3:ST-E-S2S
4:PT-S2S

2.0

Averaged Reward

For each user, 5 dialogues are collected for training and 200
dialogues are collected for testing. In the training and testing processes, all interior ground truth information and slot
information will not be used.
We use the BLEU (Papineni et al. 2002) and slot-errorrate as the off-line evaluation metric, and use the averaged reward and averaged success rate as the online evaluation metrics in order to fully evaluate our models. The
original slot-error-rate is the ratio of the number of wrongslots and missing-slots over the total number of slots in the
given ground truth dialogue act. However, in our setting, the
ground truth dialogue act is not given before response generation, thus the missing-slots do not make sense. We modify
the definition of slot-error-rate so that it can adapt to our
setting. The wrong-slots are defined as the slots not in the
simulated user’s preference. Hence we define the slot-errorrate as the ratio of the number of wrong-slots over the total number of slots in the generated sentence. Note that a
low slot-error-rate doesn’t necessarily mean the responses
are fluent or appropriate, it only means fewer wrong-slots
in the responses. For online testing, each simulated user will
generate 1000 coffee orders and the mean and standard deviation of rewards obtained in the dialogues are reported. We
run the experiments with 5 different random seeds and we
report the mean and standard deviation of the BLEU, slot
error rate, averaged rewards and success rate.
All results are shown in Figures 4-7 with details listed in
Table 2. The none-transfer models, i.e., “S2S” and “HRED”,
work well because the simulated dialogues are relatively
simple. The sentence-level transfer methods “ST-S2S”, “STE-S2S” and “ST-HRED” perform worst possibly due to
the negative transfer. By analyzing the online evaluation
records, we found that the sentence-level transfer methods
suffer from, for example, using the wrong personal information as shown in the case study section. Personalized phraselevel transfer methods obtain most of the personal information correctly and hence achieve the highest averaged reward, success rate and BLEU scores. These results show that
our personalized decoder can improve baseline models and

1

5:HRED
6:ST-HRED
7:PT-HRED

1.5
1.0
0.5
0.0

1

2

3

4

5

6

7

Figure 6: Averaged reward for the simulation dataset
avoid negative transfer caused by domain differences.

Real-World Experiments
In this section, we conduct experiments on a real-world
dataset.
The dataset is a coffee ordering dataset collected from an
O2O service in China. The dialogues are conducted between
real customers and real waiters via instant messages. The
customers are frequent shoppers, and the waiters are specially trained. On average there are four turns of interactions in each dialogue. In this dataset, there are 52 users with
8.9 dialogues on average for each user. The statistics of this
dataset are listed in Table 1.
We also use the BLEU as the evaluation metric. For the
BLEU score, 1-gram and 2-gram are used because many
sentences are relatively short. Reward and success rate cannot be calculated because policies cannot be run live on
static data. For each dialogue turn in each testing dialogue,
we randomly generate five system responses, and we calculate the averaged BLEU score over the five responses. We
train each model with five different random seeds and report

Table 2: Experiment Results
BLEU
0.5931 ± 0.0182
0.4297 ± 0.0099
0.5856 ± 0.0123
0.6685 ± 0.0050
0.6575 ± 0.0029
0.5058 ± 0.0048
0.7193 ± 0.0148

S2S
ST-S2S
ST-E-S2S
PT-S2S
HRED
ST-HRED
PT-HRED

1:S2S
2:ST-S2S
3:ST-E-S2S
4:PT-S2S

1.4

Success Rate

1.2
1.0

Simulation
Reward
SuccessRate
0.8478 ± 0.0724
0.6667 ± 0.0302
0.4485 ± 0.0606
0.5310 ± 0.0302
1.0121 ± 0.0633
0.7691 ± 0.0285
1.1493 ± 0.1086 0.7945 ± 0.0503
1.1187 ± 0.0241
0.8095 ± 0.0112
0.7860 ± 0.1085
0.7199 ± 0.0650
1.5247 ± 0.0413 0.9587 ± 0.0164

5:HRED
6:ST-HRED
7:PT-HRED

0.8

SlotError
0.0708 ± 0.0061
0.4103 ± 0.0026
0.0627 ± 0.0048
0.0638 ± 0.0038
0.0862 ± 0.0120
0.3763 ± 0.0046
0.0405 ± 0.0051

Real data
BLEU
0.1278 ± 0.0039
0.1411 ± 0.0078
0.1513 ± 0.0046
0.1609 ± 0.0055
0.1387 ± 0.0032
0.1477 ± 0.0062
0.1588 ± 0.0093

phrase-level transfer methods “PT-HRED” and “PT-S2S”
outperform their corresponding sentence-level counterparts,
i.e., “ST-S2S”, “ST-E-S2S” and “ST-HRED”, in terms of
BLEU, which demonstrates the effectiveness of the personalized decoder. Specifically, “PT-HRED” improves “STHRED” by 7.5% in terms of BLEU. These experimental results again demonstrate that our personalized decoder improves several baseline models and alleviates the negative
transfer effect.

Case study

0.6
0.4

1

2

3

4

5

6

7

Figure 7: Averaged success rate for the simulation dataset

1:S2S
2:ST-S2S
3:ST-E-S2S
4:PT-S2S

0.20

BLEU

0.18

5:HRED
6:ST-HRED
7:PT-HRED

0.16
0.14
0.12

1

2

3

4

5

6

7

Figure 8: BLEU for the real-world dataset
the mean and standard deviation of the scores in the test set.
All results are listed in Figure 8 with details listed in
Table 2. We can see that “S2S” and “HRED” have lowest
BLEU score, because the training data from each individual user is insufficient for training a competitive model. The
sentence-level transfer methods “ST-S2S”, “ST-E-S2S” and
“ST-HRED” have higher BLEU score than none-transfer
methods, which demonstrates that transfer learning can
indeed help improve the performance. The personalized

In this section, we show a case to compare a sentence-level
transfer model and a phrase-level transfer model to see how
personalized decoder avoids negative transfer. As shown in
Table 3, we can see that the sentence-level transfer method
“ST-HRED” transfers the wrong personal information from
other domains and thereby leads to the failure. On the contrary, “PT-HRED”, the phrase-level transfer method, does
transfer the correct personal information. As a result, it successfully avoids negative transfer caused by domain differences.
Table 3: A case study by comparing the sentence-level transfer model “ST-HRED” and the phrase-level transfer model
“PT-HRED”, where the personal words are in bold.
Response Generation Comparison
User: I want a cup of coffee.
Truth Agent: Same as before, tall hot macchiato
and deliver to Building No.1, Zhengda
Wudaokou Plaza, No.1199 Minsheng
Road, Pudong District, Shanghai?
ST-HRED: Same as before, tall hot latte
and deliver to the Fiyta Building, Science
and Technology Part, Shen Zhen?
PT-HRED: Same as before, tall hot macchiato
and deliver to Building No.1, Zhengda
Wudaokou Plaza, No.1199 Minsheng
Road, Pudong District, Shanghai?

Conclusion
Training a personalized dialogue system requires a large
amount of data, which is usually unavailable from an individual user. The existing seq2seq transfer learning models operate on the entire sentence, while personalized dialogues require finer granularity. We propose a personalized decoder, which can transfer phrase-level information

between different users while keeping personalized information intact. The proposed personalized decoder can easily be
combined with many models including seq2seq and HRED
to achieve knowledge transfer. Experimental results demonstrate that the phrase-level transfer personalized models improve the BLEU over multiple sentence-level transfer baseline models. We will study dialogue transfer learning under
finer-granularity like prefix and stem in the future.

References
[Bahdanau, Cho, and Bengio 2014] Bahdanau, D.; Cho, K.;
and Bengio, Y. 2014. Neural machine translation by
jointly learning to align and translate. arXiv preprint
arXiv:1409.0473.
[Bordes and Weston 2016] Bordes, A., and Weston, J. 2016.
Learning end-to-end goal-oriented dialog. arXiv preprint
arXiv:1605.07683.
[Casanueva et al. 2015] Casanueva, I.; Hain, T.; Christensen,
H.; Marxer, R.; and Green, P. 2015. Knowledge transfer between speakers for personalised dialogue management. In
Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue.
[Gašic et al. 2013] Gašic, M.; Breslin, C.; Henderson, M.;
Kim, D.; Szummer, M.; Thomson, B.; Tsiakoulis, P.; and
Young, S. 2013. POMDP-based dialogue manager adaptation to extended domains. In Proceedings of the 14th Annual
Meeting of the Special Interest Group on Discourse and Dialogue.
[Gasic et al. 2014] Gasic, M.; Kim, D.; Tsiakoulis, P.; Breslin, C.; Henderson, M.; Szummer, M.; Thomson, B.; and
Young, S. J. 2014. Incremental on-line adaptation of
POMDP-based dialogue managers to extended domains. In
Proceedings of the 15th Annual Conference of the International Speech Communication Association, 140–144.
[Genevay and Laroche 2016] Genevay, A., and Laroche, R.
2016. Transfer learning for user adaptation in spoken dialogue systems. In Proceedings of the 2016 International
Conference on Autonomous Agents & Multiagent Systems,
975–983.
[Kingma and Ba 2014] Kingma, D., and Ba, J. 2014. Adam:
A method for stochastic optimization. arXiv preprint
arXiv:1412.6980.
[Li et al. 2016] Li, J.; Monroe, W.; Ritter, A.; Galley, M.;
Gao, J.; and Jurafsky, D. 2016. Deep reinforcement learning
for dialogue generation. arXiv preprint arXiv:1606.01541.
[Luong et al. 2015] Luong, M.-T.; Le, Q. V.; Sutskever, I.;
Vinyals, O.; and Kaiser, L. 2015. Multi-task sequence to
sequence learning. arXiv preprint arXiv:1511.06114.
[Papineni et al. 2002] Papineni, K.; Roukos, S.; Ward, T.;
and Zhu, W.-J. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics,
311–318. Association for Computational Linguistics.
[Serban et al. 2015] Serban, I. V.; Sordoni, A.; Bengio, Y.;
Courville, A.; and Pineau, J. 2015. Building end-to-end dialogue systems using generative hierarchical neural network
models. arXiv preprint arXiv:1507.04808.

[Sordoni et al. 2015] Sordoni, A.; Galley, M.; Auli, M.;
Brockett, C.; Ji, Y.; Mitchell, M.; Nie, J.-Y.; Gao, J.; and
Dolan, B. 2015. A neural network approach to contextsensitive generation of conversational responses. arXiv
preprint arXiv:1506.06714.
[Sutskever, Vinyals, and Le 2014] Sutskever, I.; Vinyals, O.;
and Le, Q. V. 2014. Sequence to sequence learning with
neural networks. In Advances in Neural Information Processing Systems 27, 3104–3112.
[Sutton and Barto 1998] Sutton, R. S., and Barto, A. G.
1998. Reinforcement learning: An introduction, volume 1.
MIT press Cambridge.
[Wen et al. 2015] Wen, T.-H.; Gasic, M.; Mrksic, N.; Su, P.H.; Vandyke, D.; and Young, S. 2015. Semantically conditioned lstm-based natural language generation for spoken
dialogue systems. arXiv preprint arXiv:1508.01745.
[Wen et al. 2016a] Wen, T.-H.; Gasic, M.; Mrksic, N.; RojasBarahona, L. M.; Su, P.-H.; Ultes, S.; Vandyke, D.; and
Young, S. 2016a. Conditional generation and snapshot learning in neural dialogue systems. arXiv preprint
arXiv:1606.03352.
[Wen et al. 2016b] Wen, T.-H.; Vandyke, D.; Mrksic, N.;
Gasic, M.; Rojas-Barahona, L. M.; Su, P.-H.; Ultes, S.;
and Young, S. 2016b. A network-based end-to-end
trainable task-oriented dialogue system. arXiv preprint
arXiv:1604.04562.
[Williams and Zweig 2016] Williams, J. D., and Zweig, G.
2016. End-to-end lstm-based dialog control optimized
with supervised and reinforcement learning. arXiv preprint
arXiv:1606.01269.
[Williams 1992] Williams, R. J. 1992. Simple statistical
gradient-following algorithms for connectionist reinforcement learning. Machine learning 8(3-4):229–256.

