Improved Neural Relation Detection for Knowledge Base Question
Answering
Mo Yu†

Wenpeng Yin? Kazi Saidul Hasan‡ Cicero dos Santos†
Bing Xiang‡ Bowen Zhou†
†
AI Foundations, IBM Research, USA
?
Center for Information and Language Processing, LMU Munich
‡
IBM Watson, USA
{yum,kshasan,cicerons,bingxia,zhou}@us.ibm.com, wenpeng@cis.lmu.de

arXiv:1704.06194v2 [cs.CL] 27 May 2017

Abstract
Relation detection is a core component of
many NLP applications including Knowledge Base Question Answering (KBQA).
In this paper, we propose a hierarchical recurrent neural network enhanced by
residual learning which detects KB relations given an input question. Our
method uses deep residual bidirectional
LSTMs to compare questions and relation names via different levels of abstraction. Additionally, we propose a simple KBQA system that integrates entity
linking and our proposed relation detector to make the two components enhance
each other. Our experimental results show
that our approach not only achieves outstanding relation detection performance,
but more importantly, it helps our KBQA
system achieve state-of-the-art accuracy
for both single-relation (SimpleQuestions)
and multi-relation (WebQSP) QA benchmarks.

1

Introduction

Knowledge Base Question Answering (KBQA)
systems answer questions by obtaining information from KB tuples (Berant et al., 2013; Yao et al.,
2014; Bordes et al., 2015; Bast and Haussmann,
2015; Yih et al., 2015; Xu et al., 2016). For an
input question, these systems typically generate a
KB query, which can be executed to retrieve the
answers from a KB. Figure 1 illustrates the process
used to parse two sample questions in a KBQA
system: (a) a single-relation question, which can
be answered with a single <head-entity, relation,
tail-entity> KB tuple (Fader et al., 2013; Yih et al.,
2014; Bordes et al., 2015); and (b) a more complex
case, where some constraints need to be handled

for multiple entities in the question. The KBQA
system in the figure performs two key tasks: (1)
entity linking, which links n-grams in questions
to KB entities, and (2) relation detection, which
identifies the KB relation(s) a question refers to.
The main focus of this work is to improve the
relation detection subtask and further explore how
it can contribute to the KBQA system. Although
general relation detection1 methods are well studied in the NLP community, such studies usually
do not take the end task of KBQA into consideration. As a result, there is a significant gap between
general relation detection studies and KB-specific
relation detection. First, in most general relation
detection tasks, the number of target relations is
limited, normally smaller than 100. In contrast, in
KBQA even a small KB, like Freebase2M (Bordes et al., 2015), contains more than 6,000 relation
types. Second, relation detection for KBQA often
becomes a zero-shot learning task, since some test
instances may have unseen relations in the training
data. For example, the SimpleQuestions (Bordes
et al., 2015) data set has 14% of the golden test
relations not observed in golden training tuples.
Third, as shown in Figure 1(b), for some KBQA
tasks like WebQuestions (Berant et al., 2013), we
need to predict a chain of relations instead of a
single relation. This increases the number of target relation types and the sizes of candidate relation pools, further increasing the difficulty of KB
relation detection. Owing to these reasons, KB relation detection is significantly more challenging
compared to general relation detection tasks.
This paper improves KB relation detection to
cope with the problems mentioned above. First, in
order to deal with the unseen relations, we propose
to break the relation names into word sequences
for question-relation matching. Second, noticing
1
In the information extraction field such tasks are usually
called relation extraction or relation classification.

Question: what episode was mike kelley the writer of
Entity Linking

Question: what tv show did grant show play on in 2008
Entity Linking

Relation Detection

Mike Kelley

episodes_written

Grant Show

?

Relation
Detection

Mike Kelley

(date) from

Mike Kelley

(American baseball
player)

2008

(b)
episodes_written

…

Love Will Find a Way

…
position_played

episodes

Grant Show

SwingTown

(American actor)

2008

USA

Knowledge
Base

?

series

starring_roles

(a)
(American television
writer/producer)

Constraint
Detection

from

Scoundrels
First baseman

…

series

2010

2011

Big Love

Figure 1: KBQA examples and its three key components. (a) A single relation example. We first identify the topic entity with
entity linking and then detect the relation asked by the question with relation detection (from all relations connecting the topic
entity). Based on the detected entity and relation, we form a query to search the KB for the correct answer “Love Will Find a
Way”. (b) A more complex question containing two entities. By using “Grant Show” as the topic entity, we could detect a chain
of relations “starring roles-series” pointing to the answer. An additional constraint detection takes the other entity “2008” as
a constraint, to filter the correct answer “SwingTown” from all candidates found by the topic entity and relation.

that original relation names can sometimes help
to match longer question contexts, we propose
to build both relation-level and word-level relation representations. Third, we use deep bidirectional LSTMs (BiLSTMs) to learn different levels
of question representations in order to match the
different levels of relation information. Finally, we
propose a residual learning method for sequence
matching, which makes the model training easier
and results in more abstract (deeper) question representations, thus improves hierarchical matching.
In order to assess how the proposed improved
relation detection could benefit the KBQA end
task, we also propose a simple KBQA implementation composed of two-step relation detection.
Given an input question and a set of candidate entities retrieved by an entity linker based on the question, our proposed relation detection model plays a
key role in the KBQA process: (1) Re-ranking the
entity candidates according to whether they connect to high confident relations detected from the
raw question text by the relation detection model.
This step is important to deal with the ambiguities normally present in entity linking results. (2)
Finding the core relation (chains) for each topic
entity2 selection from a much smaller candidate
entity set after re-ranking. The above steps are
followed by an optional constraint detection step,
when the question cannot be answered by single
relations (e.g., multiple entities in the question).
Finally the highest scored query from the above
2

Following Yih et al. (2015), here topic entity refers to
the root of the (directed) query tree; and core-chain is the
directed path of relation from root to the answer node.

steps is used to query the KB for answers.
Our main contributions include: (i) An improved relation detection model by hierarchical
matching between questions and relations with
residual learning; (ii) We demonstrate that the improved relation detector enables our simple KBQA
system to achieve state-of-the-art results on both
single-relation and multi-relation KBQA tasks.

2

Related Work

Relation Extraction Relation extraction (RE) is
an important sub-field of information extraction.
General research in this field usually works on a
(small) pre-defined relation set, where given a text
paragraph and two target entities, the goal is to
determine whether the text indicates any types of
relations between the entities or not. As a result
RE is usually formulated as a classification task.
Traditional RE methods rely on large amount of
hand-crafted features (Zhou et al., 2005; Rink and
Harabagiu, 2010; Sun et al., 2011). Recent research benefits a lot from the advancement of deep
learning: from word embeddings (Nguyen and Grishman, 2014; Gormley et al., 2015) to deep networks like CNNs and LSTMs (Zeng et al., 2014;
dos Santos et al., 2015; Vu et al., 2016) and attention models (Zhou et al., 2016; Wang et al., 2016).
The above research assumes there is a fixed
(closed) set of relation types, thus no zero-shot
learning capability is required. The number
of relations is usually not large: The widely
used ACE2005 has 11/32 coarse/fine-grained relations; SemEval2010 Task8 has 19 relations; TAC-

KBP2015 has 74 relations although it considers
open-domain Wikipedia relations. All are much
fewer than thousands of relations in KBQA. As a
result, few work in this field focuses on dealing
with large number of relations or unseen relations.
Yu et al. (2016) proposed to use relation embeddings in a low-rank tensor method. However their
relation embeddings are still trained in supervised
way and the number of relations is not large in the
experiments.
Relation Detection in KBQA Systems Relation detection for KBQA also starts with featurerich approaches (Yao and Van Durme, 2014; Bast
and Haussmann, 2015) towards usages of deep
networks (Yih et al., 2015; Xu et al., 2016; Dai
et al., 2016) and attention models (Yin et al., 2016;
Golub and He, 2016). Many of the above relation detection research could naturally support
large relation vocabulary and open relation sets
(especially for QA with OpenIE KB like ParaLex
(Fader et al., 2013)), in order to fit the goal of
open-domain question answering.
Different KBQA data sets have different levels
of requirement about the above open-domain capacity. For example, most of the gold test relations
in WebQuestions can be observed during training, thus some prior work on this task adopted the
close domain assumption like in the general RE research. While for data sets like SimpleQuestions
and ParaLex, the capacity to support large relation
sets and unseen relations becomes more necessary.
To the end, there are two main solutions: (1) use
pre-trained relation embeddings (e.g. from TransE
(Bordes et al., 2013)), like (Dai et al., 2016); (2)
factorize the relation names to sequences and formulate relation detection as a sequence matching and ranking task. Such factorization works
because that the relation names usually comprise
meaningful word sequences. For example, Yin
et al. (2016) split relations to word sequences for
single-relation detection. Liang et al. (2016) also
achieve good performance on WebQSP with wordlevel relation representation in an end-to-end neural programmer model. Yih et al. (2015) use character tri-grams as inputs on both question and relation sides. Golub and He (2016) propose a generative framework for single-relation KBQA which
predicts relation with a character-level sequenceto-sequence model.
Another difference between relation detection
in KBQA and general RE is that general RE re-

search assumes that the two argument entities
are both available. Thus it usually benefits from
features (Nguyen and Grishman, 2014; Gormley
et al., 2015) or attention mechanisms (Wang et al.,
2016) based on the entity information (e.g. entity
types or entity embeddings). For relation detection in KBQA, such information is mostly missing
because: (1) one question usually contains single
argument (the topic entity) and (2) one KB entity
could have multiple types (type vocabulary size
larger than 1,500). This makes KB entity typing
itself a difficult problem so no previous used entity information in the relation detection model.3

3

Background: Different Granularity in
KB Relations

Previous research (Yih et al., 2015; Yin et al.,
2016) formulates KB relation detection as a sequence matching problem. However, while the
questions are natural word sequences, how to represent relations as sequences remains a challenging problem. Here we give an overview of two
types of relation sequence representations commonly used in previous work.
(1) Relation Name as a Single Token (relationlevel). In this case, each relation name is treated
as a unique token. The problem with this approach is that it suffers from the low relation coverage due to limited amount of training data, thus
cannot generalize well to large number of opendomain relations. For example, in Figure 1, when
treating relation names as single tokens, it will be
difficult to match the questions to relation names
“episodes written” and “starring roles” if these
names do not appear in training data – their relation embeddings hr s will be random vectors thus
are not comparable to question embeddings hq s.
(2) Relation as Word Sequence (word-level). In
this case, the relation is treated as a sequence of
words from the tokenized relation name. It has
better generalization, but suffers from the lack
of global information from the original relation
names. For example in Figure 1(b), when doing
only word-level matching, it is difficult to rank the
target relation “starring roles” higher compared
to the incorrect relation “plays produced”. This
is because the incorrect relation contains word
“plays”, which is more similar to the question
3
Such entity information has been used in KBQA systems
as features for the final answer re-rankers.

Relation Token
relation-level
word-level

episodes written
episodes
written

Question 1
what tv episodes were <e> the writer of
tv episodes were <e> the writer of
tv episodes
the writer of

Question 2
what episode was written by <e>
episode was written by <e>
episode
written

Table 1: An example of KB relation (episodes written) with two types of relation tokens (relation names
and words), and two questions asking this relation. The topic entity is replaced with token <e> which
could give the position information to the deep networks. The italics show the evidence phrase for each
relation token in the question.
(containing word “play”) in the embedding space.
On the other hand, if the target relation co-occurs
with questions related to “tv appearance” in training, by treating the whole relation as a token (i.e.
relation id), we could better learn the correspondence between this token and phrases like “tv
show” and “play on”.
The two types of relation representation contain different levels of abstraction. As shown
in Table 1, the word-level focuses more on local information (words and short phrases), and
the relation-level focus more on global information (long phrases and skip-grams) but suffer from
data sparsity. Since both these levels of granularity have their own pros and cons, we propose
a hierarchical matching approach for KB relation
detection: for a candidate relation, our approach
matches the input question to both word-level and
relation-level representations to get the final ranking score. Section 4 gives the details of our proposed approach.

4

Improved KB Relation Detection

This section describes our hierarchical sequence
matching with residual learning approach for relation detection. In order to match the question to
different aspects of a relation (with different abstraction levels), we deal with three problems as
follows on learning question/relation representations.
4.1

Relation Representations from Different
Granularity

We provide our model with both types of relation representation: word-level and relationlevel. Therefore, the input relation becomes r =
word } ∪ {r rel , · · · , r rel }, where the
{r1word , · · · , rM
1
M2
1
first M1 tokens are words (e.g. {episode, written}), and the last M2 tokens are relation names,
e.g., {episode written} or {starring roles, series}
(when the target is a chain like in Figure 1(b)).
We transform each token above to its word embed-

ding then use two BiLSTMs (with shared parameters) to get their hidden representations [Bword
1:M1 :
Brel
]
(each
row
vector
β
is
the
concatenai
1:M2
tion between forward/backward representations at
i). We initialize the relation sequence LSTMs
with the final state representations of the word sequence, as a back-off for unseen relations. We apply one max-pooling on these two sets of vectors
and get the final relation representation hr .
4.2

Different Abstractions of Questions
Representations

From Table 1, we can see that different parts of a
relation could match different contexts of question
texts. Usually relation names could match longer
phrases in the question and relation words could
match short phrases. Yet different words might
match phrases of different lengths.
As a result, we hope the question representations could also comprise vectors that summarize various lengths of phrase information (different levels of abstraction), in order to match relation representations of different granularity. We
deal with this problem by applying deep BiLSTMs on questions. The first-layer of BiLSTM
works on the word embeddings of question words
q = {q1 , · · · , qN } and gets hidden representations
(1)
(1)
(1)
Γ1:N = [γ1 ; · · · ; γN ]. The second-layer BiL(1)
STM works on Γ1:N to get the second set of hid(2)
den representations Γ1:N . Since the second BiLSTM starts with the hidden vectors from the first
layer, intuitively it could learn more general and
abstract information compared to the first layer.
Note that the first(second)-layer of question representations does not necessarily correspond to the
word(relation)-level relation representations, instead either layer of question representations could
potentially match to either level of relation representations. This raises the difficulty of matching
between different levels of relation/question representations; the following section gives our proposal to deal with such problem.

(cosine similarity)

Question
Representation

Relation
Representation

max-pooling

Point-wise
summation

…
.

…

𝜸𝒊𝟐

… Bi-LSTM 2
max-pooling

Shortcut
connections

...
…

what

tv

show

did

...

…

𝜸. 𝒊𝟏

<e>

Word-Level

Relation-Level

Bi-LSTM 1

…

Question

…

𝜷𝒓𝒌

𝜷𝒘
𝒋

starring_role series

starring

role

series

Relation

Figure 2: The proposed Hierarchical Residual BiLSTM (HR-BiLSTM) model for relation detection.
Note that without the dotted arrows of shortcut connections between two layers, the model will only
compute the similarity between the second-layer of questions representations and the relation, thus is not
doing hierarchical matching.
4.3

Hierarchical Matching between Relation
and Question

Now we have question contexts of different
(1)
(2)
lengths encoded in Γ1:N and Γ1:N . Unlike the
standard usage of deep BiLSTMs that employs
the representations in the final layer for prediction,
here we expect that two layers of question representations can be complementary to each other and
both should be compared to the relation representation space (Hierarchical Matching). This is important for our task since each relation token can
correspond to phrases of different lengths, mainly
because of syntactic variations. For example in Table 1, the relation word written could be matched
to either the same single word in the question or a
much longer phrase be the writer of.
We could perform the above hierarchical matching by computing the similarity between each
layer of Γ and hr separately and doing the
(weighted) sum between the two scores. However this does not give significant improvement
(see Table 2). Our analysis in Section 6.2 shows
that this naive method suffers from the training
difficulty, evidenced by that the converged training loss of this model is much higher than that
of a single-layer baseline model. This is mainly
because (1) Deep BiLSTMs do not guarantee that
the two-levels of question hidden representations
are comparable, the training usually falls to local
optima where one layer has good matching scores
and the other always has weight close to 0. (2)

The training of deeper architectures itself is more
difficult.
To overcome the above difficulties, we adopt the
idea from Residual Networks (He et al., 2016) for
hierarchical matching by adding shortcut connections between two BiLSTM layers. We proposed
two ways of such Hierarchical Residual Match(1)
(2)
ing: (1) Connecting each γi and γi , resulting
0
(1)
(2)
in a γi = γi + γi for each position i. Then the
final question representation hq becomes a max0
pooling over all γi s, 1≤i≤N . (2) Applying max(1)
(2)
(1)
(2)
pooling on Γ1:N and Γ1:N to get hmax and hmax ,
(1)
(2)
respectively, then setting hq = hmax + hmax . Finally we compute the matching score of r given q
as srel (r; q) = cos(hr , hq ).
Intuitively, the proposed method should benefit
from hierarchical training since the second layer is
fitting the residues from the first layer of matching,
so the two layers of representations are more likely
to be complementary to each other. This also ensures the vector spaces of two layers are comparable and makes the second-layer training easier.
During training we adopt a ranking loss to maximizing the margin between the gold relation r+
and other relations r− in the candidate pool R.
lrel = max{0, γ − srel (r+ ; q) + srel (r− ; q)}
where γ is a constant parameter. Fig 2 summarizes the above Hierarchical Residual BiLSTM
(HR-BiLSTM) model.

Remark: Another way of hierarchical matching
consists in relying on attention mechanism, e.g.
(Parikh et al., 2016), to find the correspondence
between different levels of representations. This
performs below the HR-BiLSTM (see Table 2).

Sections 5.1 and 5.2 elaborate how our relation
detection help to re-rank entities in the initial entity linking, and then those re-ranked entities enable more accurate relation detection. The KBQA
end task, as a result, benefits from this process.

5

5.1

KBQA Enhanced by Relation
Detection

This section describes our KBQA pipeline system.
We make minimal efforts beyond the training of
the relation detection model, making the whole
system easy to build.
Following previous work (Yih et al., 2015; Xu
et al., 2016), our KBQA system takes an existing
entity linker to produce the top-K linked entities,
ELK (q), for a question q (“initial entity linking”).
Then we generate the KB queries for q following
the four steps illustrated in Algorithm 1.
Algorithm 1: KBQA with two-step relation detection
Input : Question q, Knowledge Base KB, the initial
top-K entity candidates ELK (q)
Output: Top query tuple (ê, r̂, {(c, rc )})
1

2

3
4

Entity Re-Ranking (first-step relation detection): Use
the raw question text as input for a relation detector to
score all relations in the KB that are associated to the
entities in ELK (q); use the relation scores to re-rank
ELK (q) and generate a shorter list EL0K 0 (q)
containing the top-K 0 entity candidates (Section 5.1)
Relation Detection: Detect relation(s) using the
reformatted question text in which the topic entity is
replaced by a special token <e> (Section 5.2)
Query Generation: Combine the scores from step 1
and 2, and select the top pair (ê, r̂) (Section 5.3)
Constraint Detection (optional): Compute similarity
between q and any neighbor entity c of the entities
along r̂ (connecting by a relation rc ) , add the high
scoring c and rc to the query (Section 5.4).

Compared to previous approaches, the main difference is that we have an additional entity reranking step after the initial entity linking. We
have this step because we have observed that entity
linking sometimes becomes a bottleneck in KBQA
systems. For example, on SimpleQuestions the
best reported linker could only get 72.7% top-1
accuracy on identifying topic entities. This is usually due to the ambiguities of entity names, e.g. in
Fig 1(a), there are TV writer and baseball player
“Mike Kelley”, which is impossible to distinguish
with only entity name matching.
Having observed that different entity candidates
usually connect to different relations, here we propose to help entity disambiguation in the initial entity linking with relations detected in questions.

Entity Re-Ranking

In this step, we use the raw question text as input
for a relation detector to score all relations in the
KB with connections to at least one of the entity
candidates in ELK (q). We call this step relation
detection on entity set since it does not work on
a single topic entity as the usual settings. We use
the HR-BiLSTM as described in Sec. 4. For each
question q, after generating a score srel (r; q) for
each relation using HR-BiLSTM, we use the top
l best scoring relations (Rql ) to re-rank the original entity candidates. Concretely, for each entity
e and its associated relations Re , given the original entity linker score slinker , and the score of the
most confident relation r ∈ Rql ∩Re , we sum these
two scores to re-rank the entities:
srerank (e; q) =α · slinker (e; q)
+(1 − α) · max srel (r; q).
r∈Rql ∩Re

Finally, we select top K 0

< K entities according to
0
score srerank to form the re-ranked list ELK 0 (q).
We use the same example in Fig 1(a) to illustrate
the idea. Given the input question in the example, a relation detector is very likely to assign high
scores to relations such as “episodes written”,
“author of ” and “profession”. Then, according
to the connections of entity candidates in KB,
we find that the TV writer “Mike Kelley” will
be scored higher than the baseball player “Mike
Kelley”, because the former has the relations
“episodes written” and “profession”. This method
can be viewed as exploiting entity-relation collocation for entity linking.
5.2

Relation Detection

In this step, for each candidate entity e ∈
EL0K (q), we use the question text as the input to a
relation detector to score all the relations r ∈ Re
that are associated to the entity e in the KB.4 Because we have a single topic entity input in this
step, we do the following question reformatting:
we replace the the candidate e’s entity mention in
4

Note that the number of entities and the number of relation candidates will be much smaller than those in the previous step.

q with a token “<e>”. This helps the model better distinguish the relative position of each word
compared to the entity. We use the HR-BiLSTM
model to predict the score of each relation r ∈ Re :
srel (r; e, q).
5.3

Query Generation

Finally, the system outputs the <entity, relation (or
core-chain)> pair (ê, r̂) according to:
s(ê, r̂; q) =

max

0

(β · srerank (e; q)

e∈ELK 0 (q),r∈Re

+(1 − β) · srel (r; e, q)) ,
where β is a hyperparameter to be tuned.
5.4

Constraint Detection

Similar to (Yih et al., 2015), we adopt an additional constraint detection step based on text
matching. Our method can be viewed as entitylinking on a KB sub-graph. It contains two steps:
(1) Sub-graph generation: given the top scored
query generated by the previous 3 steps5 , for each
node v (answer node or the CVT node like in Figure 1(b)), we collect all the nodes c connecting to
v (with relation rc ) with any relation, and generate
a sub-graph associated to the original query. (2)
Entity-linking on sub-graph nodes: we compute
a matching score between each n-gram in the input
question (without overlapping the topic entity) and
entity name of c (except for the node in the original query) by taking into account the maximum
overlapping sequence of characters between them
(see Appendix A for details and B for special rules
dealing with date/answer type constraints). If the
matching score is larger than a threshold θ (tuned
on training set), we will add the constraint entity c
(and rc ) to the query by attaching it to the corresponding node v on the core-chain.

6
6.1

Experiments
Task Introduction & Settings

We use the SimpleQuestions (Bordes et al., 2015)
and WebQSP (Yih et al., 2016) datasets. Each
question in these datasets is labeled with the gold
semantic parse. Hence we can directly evaluate
relation detection performance independently as
well as evaluate on the KBQA end task.
5
Starting with the top-1 query suffers more from error
propagation. However we still achieve state-of-the-art on WebQSP in Sec.6, showing the advantage of our relation detection model. We leave in future work beam-search and feature
extraction on beam for final answer re-ranking like in previous research.

SimpleQuestions (SQ): It is a single-relation
KBQA task. The KB we use consists of a Freebase
subset with 2M entities (FB2M) (Bordes et al.,
2015), in order to compare with previous research.
Yin et al. (2016) also evaluated their relation extractor on this data set and released their proposed
question-relation pairs, so we run our relation detection model on their data set. For the KBQA
evaluation, we also start with their entity linking
results6 . Therefore, our results can be compared
with their reported results on both tasks.
WebQSP (WQ): A multi-relation KBQA task.
We use the entire Freebase KB for evaluation
purposes. Following Yih et al. (2016), we use
S-MART (Yang and Chang, 2015) entity-linking
outputs.7 In order to evaluate the relation detection models, we create a new relation detection
task from the WebQSP data set.8 For each question and its labeled semantic parse: (1) we first
select the topic entity from the parse; and then (2)
select all the relations and relation chains (length
≤ 2) connected to the topic entity, and set the corechain labeled in the parse as the positive label and
all the others as the negative examples.
We tune the following hyper-parameters on development sets: (1) the size of hidden states for
LSTMs ({50, 100, 200, 400})9 ; (2) learning rate
({0.1, 0.5, 1.0, 2.0}); (3) whether the shortcut
connections are between hidden states or between
max-pooling results (see Section 4.3); and (4) the
number of training epochs.
For both the relation detection experiments and
the second-step relation detection in KBQA, we
have entity replacement first (see Section 5.2
and Figure 1). All word vectors are initialized
with 300-d pretrained word embeddings (Mikolov
et al., 2013). The embeddings of relation names
are randomly initialized, since existing pre-trained
relation embeddings (e.g. TransE) usually support
limited sets of relation names. We leave the usage
of pre-trained relation embeddings to future work.
6.2

Relation Detection Results

Table 2 shows the results on two relation detection tasks. The AMPCNN result is from (Yin
et al., 2016), which yielded state-of-the-art scores
by outperforming several attention-based meth6

The two resources have been downloaded from https:
//github.com/Gorov/SimpleQuestions-EntityLinking
7
8

https://github.com/scottyih/STAGG
The dataset is available at https://github.com/Gorov/

KBQA_RE_data.
9

For CNNs we double the size for fair comparison.

Model
AMPCNN (Yin et al., 2016)
BiCNN (Yih et al., 2015)
BiLSTM w/ words
BiLSTM w/ relation names
Hier-Res-BiLSTM (HR-BiLSTM)
w/o rel name
w/o rel words
w/o residual learning (weighted sum on two layers)
replacing residual with attention (Parikh et al., 2016)
single-layer BiLSTM question encoder
replacing BiLSTM with CNN (HR-CNN)

Relation Input Views
words
char-3-gram
words
rel names
words + rel names
words
rel names
words + rel names
words + rel names
words + rel names
words + rel names

Accuracy
SimpleQuestions WebQSP
91.3
90.0
77.74
91.2
79.32
88.9
78.96
93.3
82.53
91.3
81.69
88.8
79.68
92.5
80.65
92.6
81.38
92.8
78.41
92.9
79.08

Table 2: Accuracy on the SimpleQuestions and WebQSP relation detection tasks (test sets). The top
shows performance of baselines. On the bottom we give the results of our proposed model together with
the ablation tests.
ods. We re-implemented the BiCNN model from
(Yih et al., 2015), where both questions and relations are represented with the word hash trick on
character tri-grams. The baseline BiLSTM with
relation word sequence appears to be the best baseline on WebQSP and is close to the previous best
result of AMPCNN on SimpleQuestions. Our proposed HR-BiLSTM outperformed the best baselines on both tasks by margins of 2-3% (p < 0.001
and 0.01 compared to the best baseline BiLSTM w/
words on SQ and WQ respectively).
Note that using only relation names instead
of words results in a weaker baseline BiLSTM
model. The model yields a significant performance drop on SimpleQuestions (91.2% to
88.9%). However, the drop is much smaller on
WebQSP, and it suggests that unseen relations
have a much bigger impact on SimpleQuestions.
Ablation Test: The bottom of Table 2 shows ablation results of the proposed HR-BiLSTM. First,
hierarchical matching between questions and both
relation names and relation words yields improvement on both datasets, especially for SimpleQuestions (93.3% vs. 91.2/88.8%). Second, residual
learning helps hierarchical matching compared to
weighted-sum and attention-based baselines (see
Section 4.3). For the attention-based baseline,
we tried the model from (Parikh et al., 2016) and
its one-way variations, where the one-way model
gives better results10 . Note that residual learning significantly helps on WebQSP (80.65% to
10
We also tried to apply the same attention method on deep
BiLSTM with residual connections, but it does not lead to
better results compared to HR-BiLSTM. We hypothesize that
the idea of hierarchical matching with attention mechanism
may work better for long sequences, and the new advanced
attention mechanisms (Wang and Jiang, 2016; Wang et al.,
2017) might help hierarchical matching. We leave the above
directions to future work.

82.53%), while it does not help as much on SimpleQuestions. On SimpleQuestions, even removing the deep layers only causes a small drop in performance. WebQSP benefits more from residual
and deeper architecture, possibly because in this
dataset it is more important to handle larger scope
of context matching.
Finally, on WebQSP, replacing BiLSTM with
CNN in our hierarchical matching framework results in a large performance drop. Yet on SimpleQuestions the gap is much smaller. We believe
this is because the LSTM relation encoder can better learn the composition of chains of relations in
WebQSP, as it is better at dealing with longer dependencies.
Analysis Next, we present empirical evidences,
which show why our HR-BiLSTM model achieves
the best scores. We use WebQSP for the analysis purposes. First, we have the hypothesis that
training of the weighted-sum model usually falls
to local optima, since deep BiLSTMs do not guarantee that the two-levels of question hidden representations are comparable. This is evidenced
by that during training one layer usually gets a
weight close to 0 thus is ignored. For example, one run gives us weights of -75.39/0.14 for
the two layers (we take exponential for the final
weighted sum). It also gives much lower training accuracy (91.94%) compared to HR-BiLSTM
(95.67%), suffering from training difficulty.
Second, compared to our deep BiLSTM with
shortcut connections, we have the hypothesis that
for KB relation detection, training deep BiLSTMs
is more difficult without shortcut connections. Our
experiments suggest that deeper BiLSTM does not
always result in lower training accuracy. In the
experiments a two-layer BiLSTM converges to
94.99%, even lower than the 95.25% achieved by a

single-layer BiLSTM. Under our setting the twolayer model captures the single-layer model as a
special case (so it could potentially better fit the
training data), this result suggests that the deep
BiLSTM without shortcut connections might suffers more from training difficulty.
Finally, we hypothesize that HR-BiLSTM is
more than combination of two BiLSTMs with
residual connections, because it encourages the
hierarchical architecture to learn different levels
of abstraction. To verify this, we replace the deep
BiLSTM question encoder with two single-layer
BiLSTMs (both on words) with shortcut connections between their hidden states. This decreases
test accuracy to 76.11%. It gives similar training
accuracy compared to HR-BiLSTM, indicating a
more serious over-fitting problem. This proves
that the residual and deep structures both contribute to the good performance of HR-BiLSTM.
6.3

KBQA End-Task Results

Table 3 compares our system with two published
baselines (1) STAGG (Yih et al., 2015), the stateof-the-art on WebQSP11 and (2) AMPCNN (Yin
et al., 2016), the state-of-the-art on SimpleQuestions. Since these two baselines are specially designed/tuned for one particular dataset, they do not
generalize well when applied to the other dataset.
In order to highlight the effect of different relation detection models on the KBQA end-task, we
also implemented another baseline that uses our
KBQA system but replaces HR-BiLSTM with our
implementation of AMPCNN (for SimpleQuestions) or the char-3-gram BiCNN (for WebQSP)
relation detectors (second block in Table 3).
Compared to the baseline relation detector (3rd
row of results), our method, which includes an improved relation detector (HR-BiLSTM), improves
the KBQA end task by 2-3% (4th row). Note that
in contrast to previous KBQA systems, our system does not use joint-inference or feature-based
re-ranking step, nevertheless it still achieves better
or comparable results to the state-of-the-art.
The third block of the table details two ablation
tests for the proposed components in our KBQA
systems: (1) Removing the entity re-ranking step
significantly decreases the scores. Since the reranking step relies on the relation detection models, this shows that our HR-BiLSTM model contributes to the good performance in multiple ways.
11

The STAGG score on SQ is from (Bao et al., 2016).

System
STAGG
AMPCNN (Yin et al., 2016)
Baseline: Our Method w/
baseline relation detector
Our Method
w/o entity re-ranking
w/o constraints
Our Method (multi-detectors)

Accuracy
SQ WQ
72.8 63.9
76.4
75.1

60.0

77.0
74.9
78.7

63.0
60.6
58.0
63.9

Table 3: KBQA results on SimpleQuestions (SQ)
and WebQSP (WQ) test sets. The numbers in
green color are directly comparable to our results
since we start with the same entity linking results.
Appendix C gives the detailed performance of the
re-ranking step. (2) In contrast to the conclusion
in (Yih et al., 2015), constraint detection is crucial
for our system12 . This is probably because our
joint performance on topic entity and core-chain
detection is more accurate (77.5% top-1 accuracy),
leaving a huge potential (77.5% vs. 58.0%) for the
constraint detection module to improve.
Finally, like STAGG, which uses multiple relation detectors (see Yih et al. (2015) for the three
models used), we also try to use the top-3 relation detectors from Section 6.2. As shown on the
last row of Table 3, this gives a significant performance boost, resulting in a new state-of-the-art result on SimpleQuestions and a result comparable
to the state-of-the-art on WebQSP.

7

Conclusion

KB relation detection is a key step in KBQA and
is significantly different from general relation extraction tasks. We propose a novel KB relation
detection model, HR-BiLSTM, that performs hierarchical matching between questions and KB relations. Our model outperforms the previous methods on KB relation detection tasks and allows our
KBQA system to achieve state-of-the-arts. For future work, we will investigate the integration of
our HR-BiLSTM into end-to-end systems. For example, our model could be integrated into the decoder in (Liang et al., 2016), to provide better sequence prediction. We will also investigate new
emerging datasets like GraphQuestions (Su et al.,
2016) and ComplexQuestions (Bao et al., 2016) to
handle more characteristics of general QA.
12
Note that another reason is that we are evaluating on accuracy here. When evaluating on F1 the gap will be smaller.

References
Junwei Bao, Nan Duan, Zhao Yan, Ming Zhou, and
Tiejun Zhao. 2016. Constraint-based question answering with knowledge graph. In Proceedings of
COLING 2016, the 26th International Conference
on Computational Linguistics: Technical Papers.
The COLING 2016 Organizing Committee, Osaka,
Japan, pages 2503–2514.
Hannah Bast and Elmar Haussmann. 2015. More accurate question answering on freebase. In Proceedings of the 24th ACM International on Conference
on Information and Knowledge Management. ACM,
pages 1431–1440.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy
Liang. 2013. Semantic parsing on Freebase from
question-answer pairs. In Proceedings of the 2013
Conference on Empirical Methods in Natural Language Processing. Association for Computational
Linguistics, Seattle, Washington, USA, pages 1533–
1544.
Antoine Bordes, Nicolas Usunier, Sumit Chopra, and
Jason Weston. 2015. Large-scale simple question
answering with memory networks. arXiv preprint
arXiv:1506.02075 .
Antoine Bordes, Nicolas Usunier, Alberto GarciaDuran, Jason Weston, and Oksana Yakhnenko.
2013. Translating embeddings for modeling multirelational data. In Advances in Neural Information
Processing Systems. pages 2787–2795.
Zihang Dai, Lei Li, and Wei Xu. 2016. Cfo: Conditional focused neural question answering with largescale knowledge bases. In Proceedings of the 54th
Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 800–810.
Cicero dos Santos, Bing Xiang, and Bowen Zhou.
2015. Classifying relations by ranking with convolutional neural networks. In Proceedings of the
53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint
Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Beijing, China, pages 626–634.
Anthony Fader, Luke S Zettlemoyer, and Oren Etzioni.
2013. Paraphrase-driven learning for open question
answering. In ACL (1). Citeseer, pages 1608–1618.
David Golub and Xiaodong He. 2016. Character-level
question answering with attention. arXiv preprint
arXiv:1604.00727 .
Matthew R. Gormley, Mo Yu, and Mark Dredze. 2015.
Improved relation extraction with feature-rich compositional embedding models. In Proceedings of
the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 1774–
1784.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian
Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition. pages
770–778.
Chen Liang, Jonathan Berant, Quoc Le, Kenneth D
Forbus, and Ni Lao. 2016. Neural symbolic machines: Learning semantic parsers on freebase with
weak supervision. arXiv preprint arXiv:1611.00020
.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing
systems. pages 3111–3119.
Thien Huu Nguyen and Ralph Grishman. 2014. Employing word representations and regularization for
domain adaptation of relation extraction. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short
Papers). Association for Computational Linguistics,
Baltimore, Maryland, pages 68–74.
Ankur Parikh, Oscar Täckström, Dipanjan Das, and
Jakob Uszkoreit. 2016. A decomposable attention
model for natural language inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association
for Computational Linguistics, Austin, Texas, pages
2249–2255.
Bryan Rink and Sanda Harabagiu. 2010. Utd: Classifying semantic relations by combining lexical and
semantic resources. In Proceedings of the 5th International Workshop on Semantic Evaluation. Association for Computational Linguistics, Uppsala, Sweden, pages 256–259.
Yu Su, Huan Sun, Brian Sadler, Mudhakar Srivatsa, Izzeddin Gur, Zenghui Yan, and Xifeng Yan.
2016. On generating characteristic-rich question
sets for qa evaluation. In Proceedings of the
2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 562–572.
https://aclweb.org/anthology/D16-1054.
Ang Sun, Ralph Grishman, and Satoshi Sekine. 2011.
Semi-supervised relation extraction with large-scale
word clustering. In Proceedings of the 49th Annual
Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Portland, Oregon, USA, pages 521–529.
Ngoc Thang Vu, Heike Adel, Pankaj Gupta, and Hinrich Schütze. 2016. Combining recurrent and convolutional neural networks for relation classification. In Proceedings of the 2016 Conference of
the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, San Diego, California, pages 534–539.

Linlin Wang, Zhu Cao, Gerard de Melo, and Zhiyuan
Liu. 2016. Relation classification via multi-level attention cnns. In Proceedings of the 54th Annual
Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for
Computational Linguistics, Berlin, Germany, pages
1298–1307.
Shuohang Wang and Jing Jiang. 2016. Learning
natural language inference with lstm. In Proceedings of the 2016 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, San Diego, California, pages 1442–1451.
http://www.aclweb.org/anthology/N16-1170.
Zhiguo Wang, Wael Hamza, and Radu Florian. 2017.
Bilateral multi-perspective matching for natural language sentences. arXiv preprint arXiv:1702.03814
.
Kun Xu, Siva Reddy, Yansong Feng, Songfang Huang,
and Dongyan Zhao. 2016. Question answering on
freebase via relation extraction and textual evidence.
In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1:
Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 2326–2336.
Yi Yang and Ming-Wei Chang. 2015. S-mart: Novel
tree-based structured learning algorithms applied to
tweet entity linking. In Proceedings of the 53rd Annual Meeting of the Association for Computational
Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1:
Long Papers). Association for Computational Linguistics, Beijing, China, pages 504–513.
Xuchen Yao, Jonathan Berant, and Benjamin
Van Durme. 2014.
Freebase qa: Information
extraction or semantic parsing? ACL 2014 page 82.
Xuchen Yao and Benjamin Van Durme. 2014. Information extraction over structured data: Question answering with freebase. In ACL (1). Citeseer, pages
956–966.
Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and
Jianfeng Gao. 2015. Semantic parsing via staged
query graph generation: Question answering with
knowledge base. In Association for Computational
Linguistics (ACL).
Wen-tau Yih, Xiaodong He, and Christopher Meek.
2014. Semantic parsing for single-relation question answering. In Proceedings of the 52nd Annual Meeting of the Association for Computational
Linguistics (Volume 2: Short Papers). Association
for Computational Linguistics, Baltimore, Maryland, pages 643–648.
Wen-tau Yih, Matthew Richardson, Chris Meek, MingWei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question

answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics
(Volume 2: Short Papers). Association for Computational Linguistics, Berlin, Germany, pages 201–206.
Wenpeng Yin, Mo Yu, Bing Xiang, Bowen Zhou, and
Hinrich Schütze. 2016. Simple question answering
by attentive convolutional neural network. In Proceedings of COLING 2016, the 26th International
Conference on Computational Linguistics: Technical Papers. The COLING 2016 Organizing Committee, Osaka, Japan, pages 1746–1756.
Mo Yu, Mark Dredze, Raman Arora, and Matthew R.
Gormley. 2016. Embedding lexical features via lowrank tensors. In Proceedings of the 2016 Conference
of the North American Chapter of the Association
for Computational Linguistics: Human Language
Technologies. Association for Computational Linguistics, San Diego, California, pages 1019–1029.
http://www.aclweb.org/anthology/N16-1117.
Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou,
and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings of
COLING 2014, the 25th International Conference
on Computational Linguistics: Technical Papers.
Dublin City University and Association for Computational Linguistics, Dublin, Ireland, pages 2335–
2344.
GuoDong Zhou, Jian Su, Jie Zhang, and Min Zhang.
2005. Exploring various knowledge in relation extraction. In Association for Computational Linguistics. pages 427–434.
Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen
Li, Hongwei Hao, and Bo Xu. 2016. Attentionbased bidirectional long short-term memory networks for relation classification. In Proceedings of
the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).
Association for Computational Linguistics, Berlin,
Germany, pages 207–212.

Appendix A: Detailed Score Computation
for Constraint Detection
Given an input question q and an entity name e in
KB, we denote the lengths of the question and the
entity name as |q| and |ne |. For a mention m of
the entity e which is an n-gram in q, we compute
the longest consecutive common sub-sequence between m and e, and denote its length as |m ∩ e|.
All the lengths above are measured by the number
of characters.
Based on the above numbers we compute the
proportions of the length of the overlap between
entity mention and entity name (in characters) in
|m∩e|
the entity name |m∩e|
|e| and in the question |q| ;
The final score for the question has a mention linking to e is
|m ∩ e| |m ∩ e|
slinker (e; q) = max
+
m
|q|
|e|

Appendix B: Special Rules for Constraint
Detection
1. Special threshold for date constraints. The
time stamps in KB usually follow the yearmonth-day format, while the time in WebQSP are usually years. This makes the overlap between the date entities in questions and
the KB entity names smaller (length of overlap is usually 4). To deal with this, we only
check whether the dates in questions could
match the years in KB, thus have a special
threshold of θ = 1 for date constraints.
2. Filtering the constraints for answer nodes.
Sometimes the answer node could connect
to huge number of other nodes, e.g. when
the question is asking for a country and we
have an answer candidate the U.S.. From
the observation on the WebQSP datasets,
we found that for most of the time, the
gold constraints on answers are their entity types (e.g., whether the question is asking for a country or a city). Based on
this observation, in the constraint detection
step, for the answer nodes we only keep
the tuples with type relations (i.e.
the
relation name contains the word “type”),
such as common.topic.notable types, education.educational institution.school type etc.

Appendix C: Effects of Entity Re-Ranking
on SimpleQuestions
Removing entity re-ranking step results in significant performance drop (see Table 3, the row of w/o
entity re-ranking). Table 4 evaluates our re-ranker
as an separate task. Our re-ranker results in large
improvement, especially when the beam sizes are
smaller than 10. This is indicating another important usage of our proposed improved relation detection model on entity linking re-ranking.
Top K
1
10
20
50
Top K
1
10
20
50

FreeBase API
40.9
64.3
69.7
75.7
(Yin et al., 2016)
72.7
86.9
88.4
90.2

(Golub & He, 2016)
52.9
74.0
77.8
82.0
Our Re-Ranker
79.0
89.5
90.9
92.5

Table 4: Entity re-ranking on SimpleQuestions
(test set).

