A retrieval-based dialogue system utilizing utterance
and context embeddings
Alexander Bartl and Gerasimos Spanakis
Department of Data Science and Knowledge Engineering
Maastricht University
6200MD, Netherlands
Email: a.bartl@student.maastrichtuniversity.nl,jerry.spanakis@maastrichtuniversity.nl

arXiv:1710.05780v3 [cs.CL] 20 Oct 2017

1

Abstract—Finding
semantically
rich
and
computerunderstandable representations for textual dialogues, utterances
and words is crucial for dialogue systems (or conversational
agents), as their performance mostly depends on understanding
the context of conversations. Recent research aims at finding
distributed vector representations (embeddings) for words, such
that semantically similar words are relatively close within the
vector-space. Encoding the “meaning” of text into vectors is
a current trend, and text can range from words, phrases and
documents to actual human-to-human conversations. In recent
research approaches, responses have been generated utilizing
a decoder architecture, given the vector representation of the
current conversation. In this paper, the utilization of embeddings
for answer retrieval is explored by using Locality-Sensitive
Hashing Forest (LSH Forest), an Approximate Nearest Neighbor
(ANN) model, to find similar conversations in a corpus and rank
possible candidates. Experimental results on the well-known
Ubuntu Corpus (in English) and a customer service chat dataset
(in Dutch) show that, in combination with a candidate selection
method, retrieval-based approaches outperform generative ones
and reveal promising future research directions towards the
usability of such a system.
Index Terms—Dialogue Systems, Deep Learning, Information
Retrieval

I. I NTRODUCTION
Text-only based Dialogue systems, also called Conversational Agents, Chatbots or Chatterbots, have become very
popular in the research community and for large companies.
The reason for the rise in popularity lies in the fact that
their ability to interact intelligently with humans has improved
significantly due to advancements in hardware technologies
and Artificial Intelligence.
One of the latest effective approaches [1] is to represent
words, phrases, or even complete dialogues as fixed-length
vectors of floating point numbers, also called embeddings
(or distributed representations or feature vectors). The Hierarchical Recurrent Encoder-Decoder (HRED) [2] and its
successors [3], (as well as similar related models) are specifically designed to encode the meaning of textual conversations
regarding the special structure that originates from multiple
turn-taking speakers.
1 A version of this paper is accepted at ICMLA2017 conference
http://www.icmla-conference.org/icmla17/

In our approach, a context embedding, a vector encoding the
meaning of a conversation up to a certain time step t, encoded
by the HRED model, serves as input to the decoder component
to generate a textual answer. We explore the performance of
a retrieval-based model that uses the utterance- and contextembeddings, previously generated by the HRED model, to find
similar conversations and rank possible candidate answers. We
argue that a retrieval-based approach, based on embeddings,
can outperform the generative approach, as the retrieval of
similar conversations is less dependent on high quality embeddings and less susceptible to poorly trained embeddings.
The rest of the paper is organized as follows: We first give
an outline of the research around dialogue systems, showing
the development from purely script-based systems to deep networks generating answers end-to-end. The proposed pipeline
is discussed in Section III. Experimental setup (datasets,
evaluation metrics, models implemented and compared) as
well as the results are discussed in Section IV. Finally, we
conclude the paper by summarizing the main findings and
outline future work.
II. R ELATED W ORK
The purpose of Dialogue Systems (DS), often also termed
Conversational Agents (CA) or Chatterbots, is to converse
with humans to provide information, help in decision making,
perform administrative services, or just for the sake of entertainment [4]. The traditional design of Dialogue Systems [5]
follows a modular approach, splitting the system usually into a
Natural Language Understanding (NLU) module, a Dialogue
Manager and a Natural Language Generation (NLG) unit. The
NLU module processes the raw user input and extracts useful
information and features that can be used by the Dialogue
Manager to update internal states, send queries to a knowledge
base or, more generally, find actions based on a script. The
NLG acts inversely to the NLU module, receiving features and
information from the Dialogue Manager to generate a response
that, finally, will be presented to the user.
One of the simplest design approaches for an NLU is
to simply spot certain key-words or combinations of them.
This is often the general procedure of script-based chatbots
and the approach followed by ELIZA. However, there is a
long history of attempts [6] to improve NLU and find better
representations of text. With advances in machine learning,

the development ranges from statistical modelling of language
[7], semantic parsing [8], skip-gram models [1], and others, to
approaches utilizing deep neural architectures [9], [10]. Neural
networks have also been used to improve NLG [11], [12].
Recently, Dialogue Managers have made similar advances
towards automated solutions, with a focus on reinforcement
learning [13], [14], generating policies of how to interact with
humans, based on some state representation.
With the rise of Deep Learning (DL) in recent years [15]
and an increasing company interest in chatterbots, end-to-end
Dialogue Systems, such as deep Recurrent Neural Networks,
constituting all modules in one model [2], have become one
of the major research topics for Dialogue Systems. The tasks
of NLU, NLG, and the Dialogue Manager are performed by a
single deep network that is trained to reproduce conversations
from a large dataset. Such a system would generate an answer
end-to-end from raw user input. Even though training deep
RNNs can be considerably difficult [16], only one model
would need to be optimized, and one could benefit from the
neural model’s capability to generate natural responses [17],
[11].
Our proposed pipeline is a combination of a generativeand retrieval-based approach. An encoder model, such as
the HRED model (or could be one of its more advanced
variations) is trained end-to-end on a textual corpus, using an
objective function that is based on how capable the model is
of generating the answers in the training set. After the training
however, the decoder component of the HRED model is not
used to generate answers. Instead, we argue that a retrievalbased approach taking over the NLG part performs equally or
better in both general and specific domains.
III. M ODEL DESCRIPTION
The proposed model can be split up into three individual components. The first component, the encoder, utilizes
the HRED model to encode raw conversations into embeddings containing the actual meaning. The second component,
a retrieval-based approach using an Approximate Nearest
Neighbor (ANN) model, is responsible for retrieving similar conversations from a database of embedding- and rawtext-tuples. Given the context of an unfinished conversation,
suitable responses are considered to be contained in similar
conversations, retrieved by the ANN model. The last model
component receives a retrieved set and ranks possible answers
based on answer- and context-relevance. The entire pipeline
can be seen in Figure 1.
A. Gated Recurrent Unit (GRU)
Recurrent Neural Networks (RNNs) have been designed
to process sequential data by encoding historic information
into a hidden state. When optimizing the network to predict
future values using the current input and historic values, the
hidden state naturally becomes a new representation form
of the processed data. LSTMs, first introduced 1997, are
an attempt to increase a RNN’s capabilities to remember
long-term dependencies by replacing hidden units with more

complex memory cells, capable of controlling the information
flow in and out of the cells.
The more recently proposed GRUs are similar to LSTM
cells aiming to improve a RNN’s capabilities to remember
long-term dependencies. However, they use a different gate
design, have fewer parameters to train and come without an
additional cell state. Two gates, the reset and update gates rt
and zt , operate directly on the hidden state, i.e., the hidden
layer. Parametrized by W , U and b, while conditioned on the
current input xt and previous result yt−1 , GRU gate vectors
are computed as:
zt = ϕg (Wz xt + Uz yt−1 + bz )
rt = ϕg (Wr xt + Ur yt−1 + br ),

(1)

with ϕg being the sigmoid function. The update gate zt
combines the function of the input and forget gate by controlling how much the new hidden state (ht ) is defined by
either the current input or the last hidden state, using linear
interpolation:
ht = zt ◦ ht−1 + (1 − zt ) ◦ h˜t ,

(2)

with h˜t being the candidate activation. The reset gate rt is
used to calculate h˜t , controlling similarly how much of the
previous hidden state to keep:
h˜t = ϕh (Wh xt + Uh (rt ◦ ht−1 + bh ),

(3)

with ϕh being the hyperbolic tangent function.
B. Hierarchical Recurrent Encoder-Decoder
The HRED model essentially consists of three stacked
RNNs: the utterance encoder, context encoder, and utterance
decoder, each of them depending on the result of its predecessor and operating on distributed representations.
Formally, a dialogue D, the input to such a model,
can be represented as a sequence of utterances D =
(U1 , ..., UM ), with Um being a sequence of word indices
Um = (wm,1 , ..., wm,Nm ), each of them usually pointing to a
vocabulary reference or directly to a word embedding. These
become the input to the utterance encoder.
1) Encoding steps: To better capture long-term dependencies, the GRU gating function is used for the individual RNNs.
For a simplified notation, the GRU can be expressed compactly
by combining equations 1, 2 and 3:
ht = GRU (ht−1 , xt ),

(4)

computing current hidden state ht , conditioned on the
previous, ht−1 and on current input xt . To comply with the
HRED notation, the utterance embedding hm,n of the current
utterance Um , including word wm,n , is calculated as:
hm,n = GRUutt (hm,n−1 , wm,n ).

(5)

Applying equation 5 consecutively on word embeddings
wm,1 , ..., wm,Nm , results in an equally-sized set of hidden
states hm,1 , ..., hm,Nm , where the last hidden state hm,Nm is

Fig. 1. A view of the pipeline implementing the proposed approach. An HRED encoder is used to generate context and response embeddings and an ANN
model builds on previous steps to retrieve similar conversations. Finally, the best candidate is selected according to answer- and context-relevance.

the summary of all words in the same utterance. As such, we
denote hm = hm,Nm to be the hidden state that represents
utterance Um .
Using this encoding approach, a set of utterances U1 , ..., UM
is encoded into hidden states h1 , ..., hM . Those are used as
input to the GRU-based context encoder, similar to how word
embeddings acted as input to the utterance encoder. As such,
context embeddings cm are a summary of utterances and
represent entire dialogues. They are computed as:
cm = GRUcon (cm,n−1 , hm ).

(6)

2) Decoding step: In addition to encoding a sequence
of embeddings into a hidden state, the decoder component
generates word probabilities over a vocabulary, given some
context U1 , ..., Um−1 and previous words wm,1 , ..., wm,n−1 .
Firstly, to condition the decoder RNN on previous utterances, the initialization of its hidden state is based on the
context encoders last hidden state cm−1 . If not designed
explicitly, context and decoder RNN usually have different
hidden state dimensionalities, which is why an additional
network layer is added to project context embeddings into the
decoder space:
dm,0 = tanh(D0 cm−1 + b0 ),

(7)

with parameters D0 and b0 and dm,0 being the decoder
RNN’s initial hidden state.
Given a set of words wm,1 , ..., wm,n−1 , having been previously generated or representing a training example, the
decoder RNN hidden state is similarly computed as it was
done for the encoder RNNs:

Using both, the hidden state dm,n−1 and word embedding
of wm,n−1 , the word embedding of current word wm,n is then
predicted as:
w(dm,n−1 , wm,n−1 ) = H0 dm,n−1 + E0 wm,n−1 + b0 ,

(9)

with the additional parameters H0 , E0 and b0 . H0 and
E0 control which part of the previous context- and word
embedding contribute to the new word embedding and how
much of that part is used.
By calculating the dot product of such generated word
embeddings with the embeddings in a vocabulary, one can
compute the similarity between prediction and existing words,
with the most similar word being the most likely one. The
actual probability of a word occurring next is based on this
similarity:
P (wm,n = v|wm,1:n−1 , U1:m−1 ) =
exp(e>
v w(dm,n−1 , wm,n−1 ))
,
PK
>
k=1 exp(ek w(dm,n−1 , wm,n−1 ))

(10)

with ev being the word embedding of word v, w the
predicted word embedding and K the vocabulary size.
By computing probabilities for each word in the vocabulary,
one can create a distribution from which words can be sampled. Pushing sampled words back into the decoder allows to
generate the next word, extending an utterance until an end-ofsequence meta token has been reached. Possible generations
can be explored using probability based search techniques such
as Beam Search [18].
C. Retrieval Model

dm,n = GRUdec (dm,n−1 , wm,n ),

(8)

processing words consecutively. The first iteration uses
the hidden state computed by equation 7 and a zero-value
embedding for wm,0 to predict the first word of an utterance.

Using the encoded corpus as a database of vectors, a Nearest
Neighbor Search (NNS) algorithm can be used to find close
embeddings in the whole set. For this purpose, an ANN
approach has been considered, as general space-partitioning

approaches, aiming to improve the NNS performance, suffer
from the curse of dimensionality [19].
Locality Sensitive Hashing (LSH) [20], [21] is an ANN
approach that uses a set of hashing functions to project similar
data points into buckets and as such, significantly restricts the
search space to the size of the bucket. For a projection, a binary
string label is constructed by applying k different hashing
functions to a single data point, where the output of such a
function is either one or zero. The desired goal of a hashing
function is to output the same label for similar data points and
differing labels for dissimilar ones. Therefore, binary string
labels that are similar, indicate that also the original data points
are similar. The string label is then used as a key to index a
bucket of similar data points, where a brute force approach
can be applied on a much smaller set. A collection of buckets
is called a hash-table and l tables constitute the entire model.
One of the main issues with the basic LSH algorithm [21] is
that choosing the optimal number of hashing functions k and
number of tables l requires one to know the most suitable value
for r, the threshold separating similar and dissimilar points.
The LSH-Forest algorithm solves this issue by allowing labels
with variable length and thus, eliminating parameter k.
Instead of linking fixed-length labels to buckets, the label
string is stored in a prefix tree, a binary tree (also called ’trie’),
in which keys are not contained within nodes but derived from
the path that leads from the root to a node.
For the case that two points are very similar or equal,
the length-limiting parameter km prevents their labels from
growing too large.
Each level of the tree is associated with a different hashing
function, sampled uniformly and with replacement from a
family of hashing functions H. Such a tree, an LSH-Tree, is
the equivalent to an LSH-based hash-table and the composition
of l trees is an LSH-Forest.
Given a query point p, finding close neighbors in a set of
LSH-Trees T1 , T2 , ..., Tl is performed in two phases. First, in
a top-down phase or descent, each tree Ti is searched for the
leaf node with the best match to the binary string label of q.
Labels are computed individually for each tree, starting from
the root and extending the label until a leaf node is reached.
Inspecting the matches from all trees, the match with the
longest prefix defines the tree-level x from which the bottomup accumulation, the second phase, begins.
D. Candidate Selection
Given a query context of an unfinished conversation, using the previously discussed LSH-Forest algorithm, one can
retrieve a candidate set from a database of encoded conversations. Candidate answers will be scored based on the matching
degree between the retrieved and the original context in terms
of question-to-question similarity or answer relevance or other
text-based features. However, the scoring functions introduced
in this section will solely be based on vector comparison
metrics, such as the cosine similarity, as text-based comparison
is less rewarding and more difficult and tedious to implement.
For the sake of clarity, the query context embedding is defined

as cq , the textual candidates as r1 , r2 , ..., rk , the context
embeddings of candidates as cr1 , cr2 , ..., crk , and the utterance
embeddings of candidates as hr1 , hr2 , ..., hrk .
1) Context Relevance: The similarity of two conversations
or the distance between a query context cq and a candidate
context crk has, intuitively, a big impact on the retrieved
answer, i.e, the more two questions are similar, the higher
the probability that the answers are similar as well. If the
cosine similarity has been chosen as the distance function
D, the labels returned by the nearest neighbor search are
already sorted by context-to-context distance. Formally, given
a candidate response rx and a query context cq , the Context
Relevance (CR) cost function is defined as:
costCR (rx ) = cossim (crx , cq ).

(11)

Fig. 2. A simple illustration of how CR is computed for a single candidate.

2) Answer Relevance: By manual inspection of near neighbors, it became apparent that the correct answer is usually
represented or almost captured in many topic-related candidates. Assuming that the most suitable topic for answering
is dominantly represented amongst candidates, responses are
ranked based on how much they capture the general topic.
Formally, the cost of a response rx is defined by the accumulated similarity between its respective embedding hrx and the
utterance embeddings of all other candidates (See Figure 3),
normalized by length k:

costAR (rx ) =

k
1X
cossim (hrx , hri )
k i=1

(12)

3) Combining Context and Answer Relevance: The problem with the previous approach is that the candidates that
are off-topic still contribute to the answer relevance cost.
Therefore, in a pre-step, according to the previously described
context relevance metric, the top n candidates are accumulated
to represent the best general answer topic. In the next step,
candidates are ranked based on their similarity to these n
responses. Formally, combined Context and Answer Relevance
(CAR) is defined as:
n

costCAR (rx ) =
with n ≤ k.

1X
cossim (hrx , hri ),
n i=1

(13)

Language
Total # of dialogues
Total # of turns
Total # of utterances
Total # of words
Avg. # of words per dialogue
Avg. # of turns per dialogue
Avg. # of words per turn
Avg. # of utterances per dialogue
Avg. # of words per utterance

Ubuntu
English
487,337
2,406,483
3,644,566
44,246,198
90.792
4.938
15.880
7.479
11.264

Vodafone
Dutch
384,897
6,571,902
10,461,677
122,325,433
317.81
17.07
18.65
27.18
11.58

TABLE I
S TATISTICS OF U BUNTU & VODAFONE C ORPUS

B. Evaluation process

Fig. 3. An image showing how the AR cost of a single candidate is accumulated by computing the cosine similarity with other candidate’s utterance
embeddings.

IV. E XPERIMENTS
A. Datasets
The first dataset we use is the Ubuntu Dialogue Corpus
which has been studied in most state of the art systems (similar
to HRED). The ubuntu dataset contains almost 1 million multiturn dialogues, with a total of over 7 million utterances and
100 million words. More information can be found in [22].
The second dataset we use is the Vodafone corpus which
is created by retrieving archived conversations of the Dutch
Vodafone online customer service. Customers having problems
with their phone, want to make contractual changes or experience other product related issues, often decide to talk with a
Vodafone service agent through an online chat platform.
Every conversation that was not clearly identified as Dutch
text was filtered out of the corpus using a port of Google’s Java
language detection implemention 2 . Furthermore, to guarantee
that the HRED model receives actual conversations for its
training, conversations that have less than 5 turns have also
been filtered out.
The original corpus contains phone numbers, addresses,
names, postal codes and other personal information. To
guarantee anonymization and also to allow enough generalization, this data has been replaced by a meta-token, e.g.
"<street_name>" or "<city>", which is considered to
be beneficial for the performance of word embeddings. This
way many more training examples will contain these general
concepts (like "<street_name>") and the model can learn
in which context a street name should appear. This is possible
because the word embeddings of such concepts are also tuned
during the training.
The final corpus was generated by using a minimal word
occurrence threshold of 10, resulting in a dictionary size of
42892 and an average of 0.435% unknowns per dialogue. The
complete statistics for both datasets can be seen in Table I.
2 http://code.google.com/p/language-detection/

A quantitative evaluation metric, the Recall@k measurement [22], has been used to compare the ranking performance
of models. Given a context, a set of n possible answers is
presented to a model, which has to rank the answers by their
likelihood of being the actual response. For a single evaluation
sample, if the correct answer is ranked to be amongst the
k best, the model succeeded. The overall performance of a
model is defined as the ratio of correctly ranked answers to
all answers, i.e., the percentage of correct answers that were
ranked to be amongst the k best.
By iterating over the conversations in the held-out test set,
an evaluation sample has been created for each individual turn
or response, with the previous turns representing the context
and the current turn or response being the ground truth. In
addition to the actual response, a single example also contains
n − 1 randomly sampled answers, which the model should
preferably rank lower than the true answer.
Each of the models, generative- or retrieval-based, receives
the context of a conversation from an evaluation sample and
has to generate or retrieve a suitable answer. The utteranceembedding of this answer is then used to compute the distance
to each of the 10 possible answers in the evaluation sample,
using the cosine similarity between utterance-embeddings. The
final ranking is based on this distance, placing similar answers
at the top.
As we wanted to have the same conditions for all models,
our ranking approach differs from the one used in [22],
where answer-embeddings have been directly predicted by an
additional network layer. Instead, we used Beam Search (using
5 beams) to generate an answer with the HRED model and
used the answer’s utterance embedding to compute the ranking
for the generative approach.
C. Results and discussion
For the Ubuntu corpus, a HRED model is trained and
then the generative approach (of the original model) and the
different candidate selection methods (as described in Section
III-D) are compared. Results can be found in Table II.
From this Table it is obvious that AR model outperforms
other candidate selection techniques as well as the generative approach. When taking into account the context of the
whole conversation, results are slightly worse which means
that answers are better predicted by focusing on each turn
individually rather than taking into account the entire context.

Model
HRED
HRED-CR
HRED-AR
HRED-CAR

R@1
34.8 ± 0.4
32.8 ± 0.3
44.1 ± 0.4
43.5 ± 0.4
TABLE

R@2
50.5 ± 0.4
47.5 ± 0.4
58.6 ± 0.4
58.0 ± 0.4
II

R@5
78.2 ± 0.3
74.1 ± 0.3
80.5 ± 0.3
80.3 ± 0.3

OVERALL RANKING PERFORMANCE OF MODELS ON THE UBUNTU
CORPUS . C ONFIDENCE INTERVALS (±95%) ARE SHOWN NEXT TO THE
AVERAGE PERFORMANCE .

For the Vodafone corpus, three HRED models have been
compared, each initialized with a different set of word embeddings in order to assess the effect of local/global context
in a language setting other than English. The first model, based
only on local domain knowledge, received word embeddings
trained with the gensim python library [23], a tool that, given
a corpus, will train word embeddings specifically for that corpus. The second model utilized word embeddings from [24],
representing global domain knowledge. The embeddings were
trained on a corpus consisting of 4 billion words, which was
automatically generated by analyzing Dutch websites. The last
model received word embeddings that are a combination of the
two previously described sets. Both contain word embeddings
with a feature-length of 320. However, the embeddings for the
last model will have a length of 420, using a concatenation
of global embeddings with 320 features and local embeddings
with 100 features.
As with the Ubuntu corpus, the generative and retrieval
based approaches (CR, AR, CAR) are compared and additionally in this setting, they are also tested upon different
embedding initialization approaches (HREDL , HREDG and
HREDLG ) An overview of the results can be found in Table
III.
As expected, for the majority of setups, models can predict
assistant responses easier than customer responses. The CAR
candidate selection method outperforms all other techniques
when predicting assistant responses. However, the performance of customer response prediction is slightly dominated
by AR. A reason for this could be that customers often reply
with new questions that might not be context related, making
answer relevance more important than context relevance.
Furthermore, it can be seen that initializing the HRED
model with word embeddings containing global domain
knowledge results in the best performance for candidate
selection approaches. However, combining global and local
domain knowledge has not led to the desired improvements.
This can be explained as follows: The computational graph
of the HRED model that defines its training also includes
tuning the word embeddings. As such, during the training, the
word embeddings are already altered to encode local domain
knowledge, even if they were only initialized with embeddings containing global domain knowledge. Adding additional
feature-length will in the worst case only add complexity to
the model.
The performance of the generative approaches, HREDL ,
HREDG , and HREDLG , are relatively similar. However,

HREDL slightly outperforms the others. This is contradictory, considering that the candidate selection methods, CR, AR
and CAR, clearly perform better on embeddings generated by
HREDG and HREDLG (See Table III). A reason for this
could be that utilizing global domain knowledge to generate an
answer is more difficult than using specific domain knowledge.
Especially for a very homogeneous (and domain specific)
corpus, giving standard answers can work better. Nonetheless,
similarity comparisons, used by the NNS approaches, could
still benefit from richer embeddings.
Table IV presents some examples of answers using the
Ubuntu Corpus using the generative approach (HRED) and
the proposed AR model. Finally, in Table V we present one
chat example from the evaluation process on Vodafone corpus
(translated in English by the authors).
V. C ONCLUSION AND F UTURE W ORK
End-to-End Dialogue Systems are relatively new and most
architectures are far away from being ready for deployment
in actual industry that most likely will require more years
of research. The architecture proposed in this paper can be
seen as a combination of end-to-end and modular Dialogue
System. The used retrieval based approach, utilizing dialogue
and utterance embeddings which were trained end-to-end, has
been shown to outperform the generative approach of the
HRED model.
More recently proposed end-to-end systems, the VHRED
model and Multiresolution Recurrent Neural Networks [25],
both being an improved version of the HRED model, are raising another question: Will one of these models outperform the
proposed retrieval based approach, even though all operate on
the same embeddings, i.e., at which point does the generative
approach benefit from the embeddings’ quality more than the
retrieval-based? Dialogue and utterance embeddings have been
only generated by one source, the HRED model. For comparison reasons, it would be interesting to explore the performance
of other encoding approaches, such as averaging over word
embeddings. Additionally, the embeddings generated by the
recently proposed VHRED model and Multiresolution Recurrent Neural Networks are expected to be of higher quality and
likely to improve the performance of the proposed approach.
However, one must not underestimate the importance of the
proposed ranking system, since it can directly be used by a
human agent as a means to assist communication with a client.
An interesting research direction arising from this paper would
be to allow human agents to affect the ranking score and
by this way providing feedback (in terms of reinforcement
learning [26]) to the system, which then might be able to rerank answers.
Another future direction is the simulation of conversations
with a tree search, by representing context embeddings as
states (tree nodes) and utterance embeddings as actions (connections between nodes). This technique is inspired by game
simulations where the desirable state is found through an
exploration/exploitation strategy. Using responses retrieved by
NNS as a set of actions, the search tree can explore possible

Predicting assistant
Predicting customer
responses
responses
Approach
Model
R@1
R@2
R@5
R@1
R@2
R@5
HREDL
31.2 ± 0.9
45.8 ± 0.8
73.4 ± 0.9
28.5 ± 0.7
39.8 ± 0.9
66.1 ± 0.8
Beam Search
HREDG
29.9 ± 0.7
44.3 ± 0.9
71.1 ± 0.6
27.1 ± 0.7
37.7 ± 0.9
63.1 ± 0.9
HREDLG
30.3 ± 1.0
44.2 ± 0.9
70.6 ± 0.7
28.9 ± 1.1
39.8 ± 1.1
64.6 ± 1.2
HREDL -CR
33.4 ± 0.7
48.0 ± 0.8
75.4 ± 0.6
32.8 ± 1.0
47.5 ± 1.0
73.5 ± 0.9
Context Relevance
HREDG -CR
34.6 ± 1.1
50.0 ± 1.0
76.2 ± 0.7
32.9 ± 0.8
47.3 ± 0.7
74.2 ± 0.8
HREDLG -CR
33.9 ± 0.9
48.8 ± 0.8
74.8 ± 0.8
32.5 ± 0.9
48.1 ± 0.7
74.8 ± 0.6
HREDL -AR
39.6 ± 0.7
55.7 ± 0.9
81.1 ± 0.7
40.0 ± 1.1
55.8 ± 1.0
80.4 ± 0.8
Answer Relevance
HREDG -AR
42.7 ± 0.9
58.5 ± 0.8
82.5 ± 0.7
41.0 ± 0.9
56.9 ± 0.7
81.5 ± 0.6
HREDLG -AR
43.0 ± 0.8
59.4 ± 0.8
82.7 ± 0.8
40.1 ± 0.9
55.7 ± 0.9
80.0 ± 0.9
HREDL -CAR
41.3 ± 0.8
57.2 ± 0.8
81.2 ± 0.5
39.9 ± 1.0
55.3 ± 1.0
79.6 ± 0.6
Context and Answer
HREDG -CAR
44.0 ± 0.7
59.8 ± 0.9
82.6 ± 0.7
40.9 ± 0.7
56.8 ± 0.7
80.6 ± 0.7
Relevance
HREDLG -CAR
43.8 ± 0.7
59.5 ± 0.8
82.6 ± 0.7
39.4 ± 0.7
55.1 ± 0.9
79.6 ± 0.6
TABLE III
OVERALL COMPARISON OF MODEL PRECISIONS ( IN %). C ONFIDENCE INTERVALS (±95%) ARE SHOWN NEXT TO THE AVERAGE PERFORMANCE .
TABLE IV
A NSWERS OF THE HRED MODEL AND THE RETRIEVAL - BASED APPROACH ( USING AR) TO A COUPLE OF QUESTIONS FROM THE U BUNTU CORPUS TEST
SET. Q UESTIONS CONSIDERED INTERESTING AND CLEAN WHERE PICKED BY THE AUTHOR .
Context
I have a netbook eou with 1gb of ram . Will ubuntu 12.04 be fine ?
eou With unity of course !

Response
Original: barely
AR: you don’t need much hardware to run gnome ,
unity perhaps yes though
HRED: that ’s a good idea ,
but it ’s not a good idea to stick with it

Hi Guys eou I need urgent help eou I’ve been trying all day to
install Ubuntu on my Mac eou yeah but I can’t find ANY solution
eou but can someon **unknown** me determining what to do ?
eou My **unknown** USB gives me kernel panic eou
not syncing

Original: Why dont you use a cdrom for
installing Ubuntu ?
AR: does your Bios support booting usb device ?
HRED: not sure what to tell you , sorry

any help folks **unknown** eou → help with what ? eou →
<**unknown** >hello folks , ok so i will try this again maybe
someone here is able to help me with my issue . running 12.04 on an hp
**unknown** laptop . when i plug my headphones sound still comes
from main speakers and my headphone , how can i make it so that when
i plug my headphone the sound will only play thru the headphones

Original: did you check out the **unknown**
link someone posted to you ?
AR: make sure you have the mixer and
sound devices installed in your kernel
HRED: that sounds like a hardware issue , right ?

Why can’t I install Ubuntu ? I burned the ISO to a disk , ran the disk on
windows , it **unknown** installed , but all I can get is the demo version .
And when I try to directly install the FULL version , it doesn’t do anything
but restart my computer . And the **unknown** won’t even work
unless I boot from CD .

my computer crashed ? how can I see the log ?

Question : is sshd installed by default with ubuntu 12.04 or do i need
to install openssh
is there a ” terminal service ” for linux
eou → X ? eou ssh ? eou

eou

i.e. not vnc

How hard will it be to upgrade from Ubuntu 12.04 to 12.10 ?

paths and score responses based on the quality of simulated
conversations.
Moreover, the candidate selection module of our proposed
pipeline reveals new opportunities for utilizing such a ranking/similarity model in other problems/domains such as recommender systems. Reviews of products, services, etc. could
be encoded using a model like HRED and then based on a
query (question) of a user, recommendation can take place by
ranking the most relevant reviews (answers).
Finally, another research direction towards the implementa-

Original: what is the ISO named ? **unknown** are you sure that ’s ” ubuntu ”?
AR: have you simply tried to burn a new cd ?
HRED: you have to use the alternate cd to install
the alternate cd
Original: look in /var/log - especially at the xorg log
AR: somewhere in /var/log
HRED: it should be in the menu . lst file
Original: normally installed by default
AR: ubuntu doesn’t install an ssh daemon by default
so yes
HRED: it should be installed by default
Original: i tried it and it was rather laggy
AR: idk it just seems lame lol ; x
HRED: this is what I was looking for
Original: update-manager will prompt you if you want to
upgrade when it ’s released and you press yes
AR: it is still there , just not installed by default
HRED: first you have to use sudo apt-get dist-upgrade
to get it to work

tion of such a dialogue system, is the utilization of additional
context information a service agent can see and that is not
contained in the conversation. Considering the size of corpora
used to train end-to-end systems (usually around 500, 000
conversations), manual annotation can be very slow and costly.
Finding an automated approach to make addresses, contractual
details and other features accessible to an end-to-end Dialogue
System is an interesting and rewarding task. Solving (some of)
the aforementioned problems will facilitate the deployment of
end-to-end Dialogue Systems in online chat service environ-

Context
<customer> my phone is damaged,
it is new, do i get a new phone? </u>
<assistant> did the phone fall? </u>
<customer> no </u>
<assistant> when did you got the phone? </u>
<customer> I had the phone in my pocket, just when taking it out
I saw that maybe it was scratched by my keys </u> <month> </u>
Actual Response
<assistant> have you got an insurance? </u>
HREDG
<assistant> I will gladly check it for you </u>
does it concern <number> ? </u>
HREDG -CAR Candidates (best three)
<assistant> you have <number> years warranty for your device.
internal device issues are covered.
if you haven’t damaged the phone yourself,
it falls under the regular warranty. </u>
do you have an insurance with us? then you are insured either way. </u>
<assistant> here they actually can check your device
and if necessary, send it in for repair. </u>
oh ... you have an insurance? </u>
<assistant> You can then send your device in for repair. </u>
if you have the device replacement service in your subscription
your device will be <unk> and you will receive a replacement device.
In case you don’t have it? </u>
then you can bring your device </u> to a Vodafone shop. </u>

TABLE V
A N EXAMPLE SHOWING THE THREE BEST RESPONSES RETRIEVED BY THE
HREDG -CAR MODEL TO A GIVEN CONTEXT. F OR COMPARISON
REASONS , THE ANSWER GENERATED BY THE HREDG MODEL AND THE
ACTUAL RESPONSE ARE SHOWN AS WELL .

ments, improving robustness, utility and customer experience.
ACKNOWLEDGEMENT
We would like to thank Harry Beckers and Marcel Overdijk
for their collaboration and support. We gratefully acknowledge
the support of QNH Consulting with the donation of the Nvidia
GTX 1070 GPU used for this research and for providing the
Vodafone dataset.
R EFERENCES
[1] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean,
“Distributed representations of words and phrases and their compositionality,” in Advances in neural information processing systems, 2013,
pp. 3111–3119.
[2] I. V. Serban, A. Sordoni, Y. Bengio, A. Courville, and J. Pineau, “Building End-To-End Dialogue Systems Using Generative Hierarchical Neural
Network Models,” in Proceedings of the Thirtieth AAAI Conference on
Artificial Intelligence. AAAI Press, 2016, pp. 3776–3783.
[3] I. V. Serban, A. Sordoni, R. Lowe, L. Charlin, J. Pineau, A. Courville,
and Y. Bengio, “A Hierarchical Latent Variable Encoder-Decoder Model
for Generating Dialogues,” arXiv preprint arXiv:1605.06069v3, 2016.
[4] A. Shawar and E. Atwell, “Chatbots: are they really useful?” in Journal
for Language Technology and Computational Linguistics, vol. 22, no. 1.
GSCL German Society for Computational Linguistics, 2007, pp. 29–49.
[5] D. Jurafsky and H. James, Speech and language processing an introduction to natural language processing, computational linguistics, and
speech. Pearson Education, 2000.
[6] M. Bates, “Models of natural language understanding,” in Proceedings
of the National Academy of Sciences of the United States of America,
vol. 92. National Academy Press, 1995, pp. 9977–9982.

[7] C. D. Manning and H. Schütze, “Foundations of statistical natural
language processing,” The MIT Press, 1999.
[8] J. Dowding, J. M. Gawron, D. Appelt, J. Bear, L. Cherny, R. Moore,
and D. Moran, “Gemini: A natural language system for spoken-language
understanding,” in Proceedings of the 31st annual meeting on Association for Computational Linguistics. Association for Computational
Linguistics, 1993, pp. 54–61.
[9] R. Collobert and J. Weston, “A unified architecture for natural language
processing: Deep neural networks with multitask learning,” in Proceedings of the 25th international conference on Machine learning. ACM,
2008, pp. 160–167.
[10] Y. LeCun, Y. Bengio, and G. E. Hinton, “Deep learning,” in Nature:
International weekly journal of science, vol. 521, no. 7553. Macmillan,
2015, pp. 436–444.
[11] O. Vinyals and Q. Le, “A neural conversational model,” in International
Conference on Machine Learning: Deep Learning Workshop, 2015.
[12] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning
with neural networks,” in Advances in neural information processing
systems, 2014, pp. 3104–3112.
[13] P. Shah, D. Hakkani-Tür, and L. Heck, “Interactive reinforcement
learning for task-oriented dialogue management,” in NIPS 2016 Deep
Learning for Action and Interaction Workshop, 2016.
[14] S. Young, M. Gašić, B. Thomson, and J. D. Williams, “POMDP-based
Statistical Spoken Dialogue Systems: a Review,” in Proceedings of the
IEEE, vol. 101, no. 5, 2013, pp. 1160–1179.
[15] J. Schmidhuber, “Deep learning in neural networks: An overview,” in
Neural networks, vol. 61. Elsevier, 2015, pp. 85–117.
[16] Y. Bengio, N. Boulanger-Lewandowski, and R. Pascanu, “Advances
in optimizing recurrent networks,” in Acoustics, Speech and Signal
Processing (ICASSP), 2013 IEEE International Conference on. IEEE,
2013, pp. 8624–8628.
[17] T.-H. Wen, M. Gašic, D. Kim, N. Mrkšic, P.-H. Su, D. Vandyke,
and S. Young, “Stochastic Language Generation in Dialogue using
Recurrent Neural Networks with Convolutional Sentence Reranking,”
in 16th Annual Meeting of the Special Interest Group on Discourse and
Dialogue, 2015, p. 275.
[18] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares,
H. Schwenk, and Y. Bengio, “Learning phrase representations using
RNN encoder-decoder for statistical machine translation,” in Proceedings of the Empiricial Methods in Natural Language Processing. Association for Computational Linguistics, 2014, pp. 1724–1734.
[19] R. Weber, H.-J. Schek, and S. Blott, “A quantitative analysis and
performance study for similarity-search methods in high-dimensional
spaces,” in Very Large Data Bases (VLDB) Conference, vol. 98, 1998,
pp. 194–205.
[20] S. Har-Peled, P. Indyk, and R. Motwani, “Approximate Nearest Neighbor: Towards Removing the Curse of Dimensionality.” in Theory of
computing, vol. 8, no. 1, 2012, pp. 321–350.
[21] M. Bawa, T. Condie, and P. Ganesan, “LSH Forest: self-tuning indexes
for similarity search,” in Proceedings of the 14th international conference on World Wide Web. ACM, 2005, pp. 651–660.
[22] R. Lowe, N. Pow, I. V. Serban, and J. Pineau, “The Ubuntu Dialogue
Corpus: A Large Dataset for Research in Unstructured Multi-Turn
Dialogue Systems,” in 16th Annual Meeting of the Special Interest Group
on Discourse and Dialogue, 2015.
[23] R. Řehůřek and P. Sojka, “Software Framework for Topic Modelling
with Large Corpora,” in Proceedings of the LREC 2010 Workshop on
New Challenges for NLP Frameworks. Valletta, Malta: ELRA, May
2010, pp. 45–50, http://is.muni.cz/publication/884893/en.
[24] S. Tulkens, C. Emmery, and W. Daelemans, “Evaluating Unsupervised
Dutch Word Embeddings as a Linguistic Resource,” in Proceedings
of the Tenth International Conference on Language Resources and
Evaluation (LREC 2016), N. C. C. Chair), K. Choukri, T. Declerck,
M. Grobelnik, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, and
S. Piperidis, Eds.
Paris, France: European Language Resources
Association (ELRA), may 2016.
[25] I. V. Serban, T. Klinger, G. Tesauro, K. Talamadupula, B. Zhou, Y. Bengio, and A. Courville, “Multiresolution Recurrent Neural Networks:
An Application to Dialogue Response Generation,” arXiv preprint
arXiv:1606.00776, 2016.
[26] J. D. Williams and G. Zweig, “End-to-end lstm-based dialog control
optimized with supervised and reinforcement learning,” arXiv preprint
arXiv:1606.01269, 2016.

