arXiv:1510.08565v3 [] 5 Nov 2015

Attention with Intention for a Neural Network
Conversation Model
Geoffrey Zweig
Kaisheng Yao ∗
Microsoft Research
Microsoft Research
gzweig@microsoft.com
kaisheny@microsoft.com
Baolin Peng
Chinese University of Hong Kong
blpeng@se.cuhk.edu.hk

Abstract
In a conversation or a dialogue process, attention and intention play intrinsic roles.
This paper proposes a neural network based approach that models the attention
and intention processes. It essentially consists of three recurrent networks. The
encoder network is a word-level model representing source side sentences. The
intention network is a recurrent network that models the dynamics of the intention
process. The decoder network is a recurrent network that produces responses
to the input from the source side. It is a language model that is dependent on
the intention and has an attention mechanism to attend to particular source side
words, when predicting a symbol in the response. The model is trained end-toend without labeling data. Experiments show that this model generates natural
responses to user inputs.

1

Introduction

A conversation process is a process of communication of thoughts through words. It may be considered as a structural process that stresses the role of purpose and processing in discourse [7].
Essentially, the discourse structure is intimately connected with two nonlinguistic notions: intention
and attention. In processing an utterance, attention explicates the processing of utterances, for example, paying attention to particular words in a sentence. On the other hand, intention is higher level
than attention and has its primary role of explaining discourse structure and coherence. Clearly, a
conversation process is inherently complicated because of the two levels of structures.
A conversation process may be cast as a sequence-to-sequence mapping task. In this task, the
source side of the conversation is from one person and the target side of the conversation is from
another person. The sequence-to-sequence mapping task includes machine translation, graphemeto-phoneme conversion, named entity tagging, etc. However, an apparent difference of a dialogue
process from these tasks is that a dialogue process involves multiple turns, whereas usually the above
tasks involve only one turn of mapping a source sequence to its target sequence.
Neural network based approaches have been successfully applied in sequence-to-sequence mapping
tasks. They have made significant progresses in machine translation [1, 6, 13], language understanding [8], and speech recognition [4]. Among those neural network-based approaches, one particular
approach, which is called encoder-decoder framework [1, 13], aims at relaxing much requirement
on human labeling.
Conversation models have been typically designed to be domain specific with much knowledge such
as rules [3, 18]. Recent methods [15] relax such requirement to some extent but their whole systems
∗
Presented at NIPS Workshop on Machine Learning for Spoken Language Understanding and Interaction
2015.

1

Table 1: An example of dialogue process.
user
agent
user
agent
user
agent
user
agent

my computer is infected
do you want to retrieve the files that was deleted?
the ones that the virus deleted , yes.
i can help you resolve the issue with our virus removal and protection service
ok
here is a link how to run system restore
thank you .
you welcome

are still trained with manual labels because of their sub-components that require so. Manual labels
are error prone and expensive. Therefore, it is appealing to train a system end-to-end without manual
labels. Recent works in [10, 12, 14] are in this approach.
In general, however, using knowledge is helpful. For example, the alignment information between
the source and target side is critical in grapheme-to-phoneme conversion [17] to outperform a strong
baseline using n-gram models [2]. In a neural network based machine translation system [6], the
alignment information is used to outperform a strong phrase-based baseline [5].
In the context of modeling conversation process, a neural network model may be built with the
knowledge of the structural information of conversation processes. In particular, the network may
incorporate the notion of intention and attention. To test this, we developed a model that consists
of three recurrent neural networks (RNNs). The source side RNN, or encoder network, encodes the
source side inputs. The target side RNN, or decoder network, uses an attention mechanism to attend
to particular words in the source side, when predicting a symbol in its response to the source side.
Importantly, this attention in the target side is conditioned on the output from an intention RNN.
This model, which has the structural knowledge of the conversation process, is trained end-to-end
without labels. We experimented with this model and observed that it generates natural responses to
user inputs.

2

Background

In the theory of discourse in [7], discourse structure is composed of three separate but related components. The first is the linguistic structure, which is the structure of the sequence of utterance.
The linguistic structure consists of segments of the discourse into which the utterances naturally
aggregate. The second structure is the intentional structure, which captures the discourse-relevant
purposes, expressed in each of the linguistic segments as well as relationships among them. The
third is the attentional state that is dynamic, and records the objects, properties, and relations that
are salient at each point of the discourse.
In many examples we observe, there are usually just one linguistic segment that consists of all the
utterances. Therefore, in the following, we consider a discourse with two structures: intention and
attention.
In the example in Table 1, there is a clear flow of intentions. The user states the problem, with the
user’s intention of conveying the problem to the agent. The agent receives the words, processes
them, and communicates back to the user. The user responds to the agent afterwards. Therefore,
the whole conversation process consists of three intentions processed sequentially. The first is the
intention of communication of the problem. The second intention is the process of resolving the
issue. The third is the intention of acknowledgment. In processing each of the intentions, the user
and the agent pay attention to particular words. For example, when resolving the issue, the agent
pays attention to words such as ”virus”.
2

Figure 1: The attention with intention (AWI) model. The model is unrolled into three turns. In
each turn has RNNs for encoder network and decoder network. Each session is represented by a
fixed-dimension vector, which is a hidden state of an intention RNN network.

3
3.1

The model
The attention with intention (AWI) model

We propose a model that attempts to represent the structural process of intentions and the associated
attentions. Figure 1 illustrates the model. It shows three layers of processing: encoder network,
intention network, and decoder network.
The encoder network has inputs from the current source side input. Because the source side in the
current turn is also dependent on the previous turn, the source side encoder network is linked with
the output from the previous target side. The encoder network creates a representation of the source
side in the current turn.
The intention network is dependent on its past state, so that it memories the history of intentions.
It therefore is a recurrent network, taking a representation of the source side in the current turn and
updating its hidden state.
The decoder is a recurrent network for language modeling that outputs symbol at each time. This
output is dependent on the current intention from the intention network. It also pays attention to
particular words in the source side.
In more details, a conversation has in totoal U turns. At turn u, a user in the source side, denoted in
(s,u)
superscript (s) , has an input sequence of ~x(s,u) = (xt
: t = 1, · · · , T ) with length T . An agent in
(t,u)
the target side, denoted in superscript (t) , responds to the user with ~y (t,u) = (yj
: j = 1, · · · J) with
length J. The proposed model is a conditional model of the target given the source, p(~y (t,u) |~x(s,u) ).
If there is no confusion, we may omit the session index u in the following.
3.2

Encoder network

The encoder network reads the input sentence ~x(s) , and converts them into a fixed-length or a variable
length representation of the source side sequence. There are many choices to encode the source side.
The approach we use is an RNN such that


(s)
(s)
(s)
ht = f xt , ht-1
(1)
(s)

(s)

where f (·) is an RNN. ht is the hidden state at time t in the source side. The initial state ht with
(t,k-1)
t = 0 is the last hidden state activity, hT
, of the decoder network in the previous turn k − 1.
(s)

(s)

One form of the output from this encoder is the last hidden state activity cT = hT . This is used
as a representation of the source side in the current turn to the intention network. The other form is
a variable-length representation, to be used in the attention model described in Sec. 3.4. A general
description of the variable length representation is as follows
(s)

ct

(s)

= q({ht , ∀t = 0, · · · T })
3

(2)

where q(·) might be a linear network or a nonlinear network.
3.3

Intention network

The signal from the encoder network is fed into an intention network to model the intention process.
Following [7], the intention process is a dynamic process to model the intrinsic dynamics of conversation, in which an intention in one turn is dependent on the intention in the previous turn. This
property might be modeled using a Markov model, but we choose an RNN.
Interestingly, the hidden state of an RNN in a certain turn may be considered as a distributed representation of the intention. Different from the usual process of training distributed representation of
words [9], the distribution representation of intentions are trained with previous turns as their context. We use a first order RNN model, in which a hidden state is dependent explicitly on its previous
state.
The intention model in AWI is therefore an RNN as follows


(s)
(t,k-1)
h(i,k) = f cT , h(i,k-1) , hT

(3)

(s)

where cT is the fixed dimension representation of the source side described in Sec. 3.2. k is the
(t,k-1)
is the last hidden layer activity of the decoder network in the
index of the current turn. hT
previous turn k − 1.
3.4

Decoder network

The last step is to decode the sequence in the target side, which is framed as a language model
over each symbol, generated left to right. In this framework, the decoder computes conditional
probability as
(t) (t)
(t)
(t)
(t) (t)
p(yj |y1 , · · · , yj-1 , ~x(s) ) = g(yj-1 , hj , cj )
(4)
where the hidden state in the decoder is computed using an RNN


(t)
(t)
(t) (t)
hj = f yj-1 , hj-1 , cj

(5)

(t)

The initial state hj at t = 0 is the last hidden state activity from the intention network.
(t)

(t)

The cj is a vector to represent the context to generate yj . It is dependent on the source side as


(t)
(t)
(s)
cj
= z hj-1 , {ct : ∀t = {1, · · · , T }}
(6)
(s)

where z(·) summerizes the variable-length source side representations {ct } using weighted average. The weight is computed using a content-based alignment model [1] that produces high scores
(s)
(t)
if the target side hidden state in previous time hj-1 and ct are similar. More formally, the weight
(s)

αjt for the context ct

is computed using
exp ejt
m exp(ejm )

αjt = P
where

(t)

(7)

(s)

ejt = a(hj-1 , ct )

(8)
(s)

The alignment model enables an attention to particular words, represented as a vector ct in the
source side. Since the decoder network generates responses on condition of the attention and also
the intention, our model is called attention with intention (AWI) model.
3.5

Implementation details

All of the recurrent networks are implemented using a recently proposed depth-gated long-short(s)
term memory (LSTM) network [16]. The context vector ct is an embedding vector of the source
side word at time t.
4

Table 2: Perplexity results with AWI model. Models have 50 or 200 hidden layer dimension, 25
alignment dimension, and one layer of LSTMs.
# hidden dimension
50
200

PPL
30.8
22.1

The alignment model in Eq. (8) follows the attention model in [1], in which ejt is calculated as


(t)
(s)
ejt = ~v > tanh W(ah) hj-1 + W(ae) ct
,
(9)
which is a neural network with one hidden layer of size A and a single output, parameterised by
W(ae) ∈ RA×H , W(ah) ∈ RA×H and ~v ∈ RA . H and A are the hidden layer dimension and
alignment dimension.

4

Evaluation

We used an in-house dialogue dataset. The dataset consists of dialogues from a helpdesk chat service. In this service, costumers seeks helps on computer related issues from human agents. Training
consists of 10000 dialogues with 96913 turns or conversations. Number of tokens is 2215047 in the
source side and 2378950 in the target side. The vocabulary size is 9085 including words from both
side. Development set data has 1000 dialogues with 9971 turns. Test set data has 500 dialogues with
5232 turns.
We use sentence-level SGD without momentum. Learning rate is initialized to 0.1. Development set
is used to control the learning rate. The learning rate is halved when perplexity on the development
is increased. One epoch of training has one pass of the training data. The order of training dialogues
is randomly shuffled in the beginning of each epoch. The order of turns in the dialogue is however
kept.
4.1

Performances measure in perplexity

An objective comparison of different models for conversation is still an open question. We report
perplexity (PPL), though it may have drawbacks, to compare different models. Table 2 presents
results in perplexity with two models with different hiden layer sizes. Results show that a larger
model with 200 hidden layer dimension has lower PPL than the model with 50 dimension.
4.2

Examples of outputs from the trained model

Table 3 lists an example of the conversation process between a human and the trained model. The
model has two layers of LSTMs and other setups are the same as used in Sec 4.1. Similarly as
observed in [11], the model produces natural responses to user inputs. The flow of intentions is
clearly seen in this example.

5

Related work

Our work is related to the recent works in [10, 12, 14], which use an encoder-decoder framework
to model conversation. The work in [10] is a model for single turn conversation. The work in [14]
is a simple encoder-decoder method using a fixed-dimension representation of the source side. The
work in [12] also uses a fixed-dimension representaiton of the source side but has an additional RNN
to model dialogue context. This additional RNN is similar to the intention RNN in AWI model.
However, AWI model differs from [12] in that it incorprates the concept of attention and intention
based on the theory in [7]. Therefore, attention mechanism is essential to AWI. The model in [12]
doesn’t have an attention model.
5

Table 3: The generated response from the trained model.
user
agent
user
agent
user
agent
user
agent
user
agent
user
agent
user
agent
user
agent
agent
user
agent
user
agent
user
agent
user
agent
user
agent
user
agent
user
agent
user
agent
agent
agent
user
agent
user
agent
user
agent

hi
how may i help you today ?
my computer is not responding
oh ! i sorry to hear that . are you getting an error message ?
yes
what recent changes did you made with this computer ?
i updated my computer
i see . is this on windows 7 ?
no it is on linux
ok . are you using the affected computer right now ?
no
i can definitely assist you on your issue .
thanks
we can go ahead and start working on this issue within our office premium software support service .
that is allright
if in case the issue comes back at all within the next 30 days we will perform the service again for free .
let me go ahead and prepare the offer and let start working on this issue now , alright ?
please
sure . may i have your phone number please .
xxx-xxx-xxxx
thank you .
no problem
thank you for waiting .
no problem
i see .
what is the next step
i see . let me go ahead and prepare the offers for you .
okay
thank you .
no problem
thank you for contacting answer desk . have a great day !
bye
thanks again for working with me . id like to send you a quick survey .
it starts with a yes/no question , with a few more questions about your experience with me today.
the survey takes about a minute and really helps us improve our service . can i send it to you now ?
sure
thank you .
no problem
thank you for contacting answer desk . have a great day !
bye
goodbye .

Because it is not yet clear what objective measure to use to compare different models, it is hard to
make claims of superiority of these models. We believe AWI model is an alternative to the models
in [12, 14].

6

Conclusions and discussions

We have presented a model that incorporates attention and intention processes in a neural network
model. Preliminary experiments show that this model generates natural responses to user inputs. Future works include experiments on common dataset to compare different models and incorporating
objective functions such as goals.
6

References
[1] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align
and translate. In Proceedings of the International Conference on Learning Representations
(ICLR), San Diego, CA, 2015.
[2] M. Bisani and H. Ney. Joint-sequence models for grapheme-to-phoneme conversion. Speech
Communication, 50(5), 2008.
[3] D. Bohus and A. I. Rudnicky. The ravenclaw dialog management framework: architecture and
systems. Computer, Speech and Language, 23:332–361, 2009.
[4] W. Chan, N. Jaitly, Q. V. Le, and O. Vinyals. Listen, attend and spell. In arXiv:1508.01211
[cs.CL], 2015.
[5] D. Chiang. Hierarchical phrase-based translation. Computational Linguistics, 33(2):201–228,
1999.
[6] J. Devlin, R. Zbib, Z. Huang, T. Lamar, R. Schwartz, and J. Makhoul. Fast and robust neural
network joint models for statistical machine translation. In ACL, 2014.
[7] B. J. Grosz and C. L. Sidner. Attention, intentions, and the structure of discourse. Computational Linguistics, 12:175–204, 1986.
[8] G. Mesnil, Y. Dauphin, K. Yao, Y. Bengio, L. Deng, D. Hakkani-Tur, X. He, L. Heck, G. Tur,
D. Yu, and G. Zweig. Using recurrent neural networks for slot filling in spoken language
understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2015.
[9] T. Mikolov, K. Chen, G. Corrado, and J. Dean. Effcient estimation of word representations in
vector space. In NIPS, 2013.
[10] L. Shang, Z. Lu, and H. Li. Neural responding machine for short-text conversation. In ACL,
2015.
[11] A. Sordoni, Y. Bengio, H. Vahabi, C. Lioma, J. G. Simonsen, and J.-Y. Nie. A hierarchical
recurrent encoder-decoder for generative context-aware query suggestion. In arXiv:1507–0222
[cd.NE], July 2015.
[12] A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji, M. Mitchell, J.-Y. Nie, J. Gao, and B. Dolan.
A neural network approach to context-sensitive generation of conversation responses. In
NAACL, 2015.
[13] I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks.
In Neural Information Processing Systems (NIPS), pages 3104–3112, Montréal, 2014.
[14] O. Vinyals and Q. V. Le. A nerual converstion model. In ICML Deep Learning Workshop,
2015.
[15] T.-H. Wen, M. Gasic, D. Kim, N. Mrksic, P.-H. Su, D. Vandyke, and S. Young. Stochastic
language generation in dialogue using recurrent neural networks with convolutional sentence
reranking. Technical report, May 2015.
[16] K. Yao, T. Cohn, E. Vylomova, K. Duh, and C. Dyer. Depth-gated LSTM. In arXiv:1508.03790
[], 2015.
[17] K. Yao and G. Zweig. Sequence-to-sequence neural net models for grapheme-to-phoneme
conversion. In INTERSPEECH, 2015.
[18] S. Young, M. Gasic, B. Thomson, and J. D. Williams. POMDP-based statistical spoken dialog
systems: A review. Proceedings of the IEEE, 101:1160–1179, 2013.

7

