DYNAMIC I NTEGRATION OF BACKGROUND
K NOWLEDGE IN N EURAL NLU S YSTEMS
Dirk Weissenborn
German Research Center for AI
dirk.weissenborn@dfki.de

Tomáš Kočiský & Chris Dyer
DeepMind
{tkocisky,cdyer}@google.com

arXiv:1706.02596v2 [cs.CL] 25 Oct 2017

A BSTRACT
Common-sense or background knowledge is required to understand natural language, but in most neural natural language understanding (NLU) systems, the
requisite background knowledge is indirectly acquired from static corpora. We
develop a new reading architecture for the dynamic integration of explicit background knowledge in NLU models. A new task-agnostic reading module provides
refined word representations to a task-specific NLU architecture by processing
background knowledge in the form of free-text statements, together with the taskspecific inputs. Strong performance on the tasks of document question answering
(DQA) and recognizing textual entailment (RTE) demonstrate the effectiveness
and flexibility of our approach. Analysis shows that our models learn to exploit
knowledge selectively and in a semantically appropriate way.

1

I NTRODUCTION

Understanding natural language depends crucially on common-sense or background knowledge, for
example, knowledge about what concepts are expressed by the words being read (lexical knowledge), and what relations hold between these concepts (relational knowledge). As a simple illustration, if an agent needs to understand that the statement “King Farouk signed his abdication” is
entailed by “King Farouk was exiled to France in 1952, after signing his resignation”, it must know
(among other things) that abdication means resignation of a king.
In most neural natural language understanding (NLU) systems, the requisite background knowledge
is implicitly encoded in the models’ parameters. That is, what background knowledge is present
has been learned from task supervision and also by pre-training word embeddings (where distributional information reliably reflects certain kinds of useful background knowledge, such as semantic
relatedness). However, acquisition of background knowledge from static training corpora is limiting
for two reasons. First, we cannot expect that all background knowledge that could be important for
solving an NLU task can be extracted from a limited amount of training data. Second, as the world
changes, the facts that may influence how a text is understood will likewise change. In short: building suitably large corpora to capture all relevant information, and keeping the corpus and derived
models up to date with changes to the world would be impractical.
In this paper, we develop a new architecture for dynamically incorporating external background
knowledge in NLU models. Rather than relying only on static knowledge implicitly present in the
training data, supplementary knowledge is retrieved from a knowledge base to assist with understanding text inputs. Since NLU systems must necessarily read and understand text inputs, our
approach incorporates background knowledge by repurposing this reading machinery—that is, we
read the text being understood together with supplementary natural language statements that assert
facts (assertions) which are relevant to understanding the content (§2).
Our knowledge-augmented NLU systems operate in a series of phases. First, given the text input
that the system must understand, which we call the context, a set of relevant supporting assertions
is retrieved. While learning to retrieve relevant information for solving NLU tasks is an important
question (Nogueira & Cho, 2017; Narasimhan et al., 2016, inter alia), in this work, we focus on
learning how to incorporate retrieved information, and use simple heuristic retrieval methods to
identify plausibly relevant background from an external knowledge base. Once the supplementary
texts have been retrieved, we use a word embedding refinement strategy that incrementally reads the
1

context and retrieved assertions starting with context-independent word embeddings and building
successively refined embeddings of the words that ultimately reflect both the relevant supporting
assertions and the input context (§3). These contextually refined word embeddings, which serve
as dynamic memory to store newly incorporated knowledge, are used in any task-specific reading
architecture. The overall architecture is illustrated in Figure 1. Although we are incorporating a
new kind of information into the NLU pipeline, a strength of this approach is that the architecture
of the reading module is independent of the final NLU task—the only requirement is that the final
architecture use word embeddings.
We carry out experiments on several different datasets on the tasks of document question answering (DQA) and recognizing textual entailment (RTE) evaluating the impact of our proposed solution with both basic task architectures and a sophisticated task architecture for RTE (§4). We find
that our embedding refinement strategy is quite effective (§5). On four standard benchmarks, we
show that refinement helps—even refining the embeddings just using the context (and no additional
background information) can improve performance significantly, and adding background knowledge
helps further. Our results are very competitive, setting a new state-of-the-art on the recent TriviaQA
benchmarks which is remarkable considering the simplicity of the chosen task-specific architecture.
Finally, we provide a detailed analysis of how knowledge is being used by an RTE system (§6),
including experiments showing that our system is capable of making appropriate counterfactual inferences when provided with “false knowledge”.

2

E XTERNAL K NOWLEDGE AS S UPPLEMENTARY T EXT I NPUTS

Knowledge resources make information that could potentially be useful for improving NLU available in a variety different formats, such as (subject, predicate, object)-triples, relational databases,
and other structured formats. Rather than tailoring our solution to a particular structured representation, we assume that all supplementary information either already exists in natural language
statements or can easily be recoded as natural language. In contrast to mapping from unstructured
to structured representations, the inverse problem is not terribly difficult. For example, given a triple
(monkey, isA, animal) we can construct the free-text assertion “a monkey is an animal” using a few
simple rules. Finally, the free-text format means that knowledge that exists only in unstructured text
form is usable by our system.
A major question that remains to be answered is: given some text that is to be understood, what supplementary knowledge should be incorporated? The retrieval of contextually relevant information
from knowledge sources is a complex research topic by itself, and it is likewise crucially dependent
on the format of the underlying knowledge base. There are several statistical (Manning et al., 1999)
and more recently neural approaches (Mitra & Craswell, 2017) and approaches based on reinforcement learning (Nogueira & Cho, 2017). In this work we make use of a simple heuristic from which
we almost exhaustively retrieve all potentially relevant assertions (see §4), and rely on our reading
architecture to learn to extract only relevant information.
In the next section, we turn to the question of how to leverage the retrieved supplementary knowledge
(encoded as text) in a NLU system.

3

R EFINING W ORD E MBEDDINGS BY R EADING

In order to incorporate information from retrieved input texts we propose to compute contextually
refined word representations prior to processing the NLU task at hand and pass them to the task
in the form of word embeddings. Word embeddings thus serve as a form of memory that not only
contains general-purpose knowledge (as in typical neural NLU systems) but also contextual information (including retrieved background knowledge). Our incremental refinement process encodes
input texts followed by updates on the word embedding matrix using the encoded input in multiple
reading steps. Words are first represented non-contextually (i.e., standard word type embeddings),
which can be conceived of as the columns in an embedding matrix E0 . At each progressive reading
step ` ≥ 1, a new embedding matrix E` is constructed by refining the embeddings from the previous step E`−1 using (user-specified) contextual information X ` for reading step `, which is a set of
natural language sequences (i.e., texts). An illustration of our incremental refinement strategy can
be found in Figure 1.
2

group
onlookers
glance
watch
people

Premise
a group of onlookers glance at a person doing a strange trick on her head

...

group
onlookers
glance
watch
people

Hypothesis
people watch another person do a trick

...
group
onlookers
glance
watch
people

Assertions
onlooker related to watches

people is a group

...

group
onlookers
glance
watch
people

...

Task
System
(RTE)

p: a group of onlookers glance at a person
doing a strange trick on her head
q: people watch another person do a trick

Entailment?

no update →
weighted update →

Figure 1: Illustration of our context-dependent, refinement strategy for word representations on an
example from the SNLI dataset comprising the premise, hypothesis and additional external information in form of free-text assertions. The reading architecture constructs refinements of word representations incrementally (conceptually represented as columns in a series of embedding matrices)
E` are incrementally refined by reading the input text and textual renderings of relevant background
knowledge before computing the representations used by the task model (in this figure, RTE).

In the following, we define this procedure formally. We denote the hidden dimensionality of our
model by n and a fully-connected layer by FC(z) = Wz + b, W ∈ Rn×m , b ∈ Rn , u ∈ Rm .
3.1

U NREFINED W ORD E MBEDDINGS (E0 )

The first representation level consists of non-contextual word representations, that is, word representations that do not depend on any input; these can be conceived of as an embedding matrix E0
whose columns are indexed by words in Σ∗ . The non-contextual word representation e0w for a single
0
word w is computed by using a gated combination of fixed, pre-trained word vectors epw ∈ Rn with
n
learned character-based embeddings echar
w ∈ R . The formal definition of this combination is given
in Eq. 1:
0

epw = ReLU(FC(epw ))

 0 
epw
gw = σ FC char
ew
e0w = gw

0

epw + (1 − gw )
3

echar
w .

(1)

We compute echar
using a single-layer convolutional neural network using n convolutional filters
w
of width 5 followed by a max-pooling operation over time. Combining pre-trained with character
based word embeddings in such a way is common practice. Our approach follows (Seo et al., 2017;
Weissenborn et al., 2017).
3.2

C ONTEXTUALLY R EFINED W ORD R EPRESENTATIONS (E` , ` ≥ 1)

In order to compute contextually refined word embeddings E` given prior representations E`−1 we
assume a given set of texts X ` = {x`1 , x`2 , . . .} that are to be read at refinement iteration `. Each text
x`i is a sequence of word tokens. We embed all tokens of every x`i using the embedding matrix from
the previous layer, E`−1 . To each word, we concatenate a one-hot vector of length L with position
` set to 1, indicating which layer is currently being processed.1 Stacking the vectors into a matrix,
`
we obtain a X`i ∈ Rd×|xi | . This matrix is processed by a bidirectional recurrent neural network, a
BiLSTM (Hochreiter & Schmidhuber, 1997) in this work. The resulting output is further projected
to X̂`i by a fully-connected layer followed by a ReLU non-linearity (Eq. 2).
X̂`i = ReLU(FC(BiLSTM(X`i )))

(2)

`−1
of word w, we initially maxpool all representations
To finally update the previous embedding ew
of occurrences matching the lemma of w in every x ∈ X ` resulting in ê`w (Eq. 3). Finally, we combine the context-independent representation ew with êw to form a context-sensitive representation
e`w via a gated addition which lets the model determine how much to revise the embedding with the
newly read information (Eq. 5).


ê`w = max x̂`k | x` ∈ X ` , lemma(x`k ) = lemma(w)

 `−1 
ew
`
uw = σ FC
ê`w
e`w = u`w

e`−1
+ (1 − u`w )
w

ê`w

(3)
(4)
(5)

Note that we soften the matching condition for w using lemmatization, lemma(w), during the pooling operation of Eq. 3 because contextual information about certain words is usually independent of
the current word form w they appear in. As a consequence, this minor linguistic pre-processing step
allows for additional interaction between tokens of the same lemma.
The important difference between our contextual refinement step and conventional multi-layer
(RNN) architectures is the pooling operation that is performed over occurrences of tokens that share
the same lemma. This effectively connects different positions within and between different texts
with each other thereby mitigating the problems arising from long-distance dependencies. More
importantly, however, it allows models to make use of additional input such as relevant background
knowledge.

4

E XPERIMENTAL S ETUP

We run experiments on four benchmarks for two popular tasks, namely recognizing textual entailment (RTE) and document question answering (DQA). In the following we describe different aspects
of our experimental setup in more detail.
Task-specific Models Our primary interest is to explore the value of our refinement strategy with
relatively generic task architectures. Therefore, we chose basic single-layer bidirectional LSTMs
(BiLSTMs) as encoders with a task-specific, feed-forward neural network on top for the final prediction. Such models are common baselines for NLU tasks and can be considered general reading
architectures as opposed to the more highly tuned, task-specific NLU systems that are necessary
1
Adding this one-hot feature lets the refinement model to learn update word embeddings differently in
different levels.

4

to achieve state-of-the art results. However, since such models frequently underperform more customized architectures, we also add our refinement module to a reimplementation of a state-of-the-art
architecture for RTE called ESIM (Chen et al., 2017b).
All models are trained end-to-end jointly with the refinement module. For the DQA baseline system
we add a simple lemma-in-question feature (liq) as suggested in Weissenborn et al. (2017) when
encoding the context to compare against competitive baseline results. We provide the exact model
implementations for our BiLSTM baselines and general training details in Appendix A.

Question Answering We apply our DQA models on 2 recent DQA benchmark datasets, SQuAD
(Rajpurkar et al., 2016) and TriviaQA (Joshi et al., 2017). The task is to predict an answer span
within a provided document p given a question q. Both datasets are large-scale, containing on the
order of 100k examples. Because TriviaQA is collected via distant supervision the test set is divided
into a large but noisy distant supervision part and a much smaller (on the order of hundreds) human
verified part. We report results on both. See Appendix A.1 for implementation details.

Recognizing Textual Entailment We test on the frequently used SNLI dataset (Bowman et al.,
2015), a collection of 570k sentence pairs, and the more recent MultiNLI dataset (433k sentence
pairs) (Williams et al., 2017). Given two sentences, a premise p and a hypothesis q, the task is to
determine whether p either entails, contradicts or is neutral to q. See Appendix A.2 for implementation details.

Knowledge Source We make use of ConceptNet2 (Speer & Havasi, 2012), a freely-available,
multi-lingual semantic network that originated from the Open Mind Common Sense project and
incorporates selected knowledge from various other knowledge sources, such as Wiktionary3 , Open
Multilingual WordNet4 , OpenCyc and DBpedia5 . It presents information in the form of relational
triples.6

Assertion Retrieval We would like to obtain information about the relations of words and phrases
between q and p from ConceptNet in order to strengthen the connection between the two sequences. Because assertions a in ConceptNet come in form of (subject, predicate, object)-triples
(s, r, o), we retrieve all assertions for which s appears in q and o appears in p, or vice versa.
Because still too many such assertions might be retrieved for an instance, we rank all retrievals
based on their respective subject and object. To this end we compute a ranking score which is
the inverse product of appearances of the subject and the object in the KB, that is score(a) =
P
P
−1
( a0 I(sa0 = sa ) · a0 I(oa0 = oa )) , where I denotes the indicator function. This is very related to the popular idf score (inverted document frequency) from information retrieval which ranks
terms higher that appear less frequently across different documents. During training and evaluation
we only retain the top-k assertions which we specify for the individual experiments separately. Note
that (although very rarely) it might happen that no assertions are retrieved at all.

Refinement Order When employing our embedding-refinement strategy, we first read the document (p) followed by the question (q) in case of DQA, and the premise (p) followed by the hypothesis (q) for RTE, that is, X 1 = {p} and X 2 = {q}. Additional knowledge in the form of a
set of assertions A is integrated after reading the task-specific input for both DQA and RTE, that
is, X 3 = A. In preliminary experiments we found that the final performance is not significantly
sensitive to the order of presentation so we decided to fix our order as defined above.
5

Model

SQuAD Dev
F1
Exact

TriviaQA Wiki Test
F1
Exact

TriviaQA Web Test
F1
Exact

BiLSTM + liq
+ reading
+ reading + knowledge (20)
+ reading + knowledge (50)

74.3
77.8
79.2
79.6

63.7
67.3
69.0
69.4

50.9 / 55.1
54.1 / 59.6
54.6 / 60.7
55.1 / 59.9

45.2 / 48.8
47.5 / 52.2
48.4 / 55.1
48.6 / 53.4

50.9 / 60.0
54.7 / 65.9
56.9 / 67.3
56.7 / 68.0

44.2 / 54.7
48.1 / 60.3
50.6 / 61.9
50.6 / 63.2

SotA Results

81.81
82.82

73.21
75.62

46.9 / 55.83
52.9 / 59.51

43.2 / 49.33
46.9 / 54.51

48.3 / 57.63
52.9 / 61.51

44.3 / 53.33
46.7 / 57.01

Table 1: Results on the SQuAD development set as well as TriviaQA-Wikipedia and -Web test
sets for n = 300−dimensional models. TriviaQA results are further divided by distant supervision
results (left) and human verified results (right). The model using external knowledge is trained with
the top-20/50 retrieved ConceptNet assertions. The liq-feature (lemma-in-question) is only used for
the baseline. 1 Hu et al. (2017), 2 Wang et al. (2017), 3 Pan et al. (2017).

5

R ESULTS

5.1

Q UESTION A NSWERING : SQ UAD AND T RIVIAQA

Table 1 presents our results on two question answering benchmarks. We report results on the SQuAD
development set7 and the two more challenging TriviaQA test sets which demonstrate that the introduction of our reading architecture helps consistently with additional gains from using background
knowledge. Our systems do even outperform current state-of-the-art models on TriviaQA which is
surprising given the simplicity of our task-specific architecture and the complexity of the others. For
instance, the system of Hu et al. (2017) uses a complex multi-hop attention mechanism to achieve
their results. Even our baseline BiLSTM + liq system reaches very competitive results on TriviaQA
which is in line with findings of Weissenborn et al. (2017). To verify that it is not the additional
computation that gives the performance boosts when using only our reading architecture (without
knowledge), we also ran experiments with 2-layer BiLSTMs (+liq) for our baselines which exhibit
similar computational complexity to BiLSTM + reading. We found that the second layer even hurts
performance. This demonstrates that pooling over word/lemma occurrences in a given context between layers, which constitutes the main difference to conventional stacked RNNs, is a powerful,
yet simple technique. In any case, the most important finding of these experiments is that knowledge
actually helps considerably with up to 2.2/2.9% improvements on the F1/Exact measures.
5.2

R ECOGNIZING T EXTUAL E NTAILMENT: SNLI AND M ULTI NLI

Table 2 shows the results of our RTE experiments. In general, the introduction of our refinement
strategy almost always helps, both with and without external knowledge. When providing additional
background knowledge from ConceptNet, our BiLSTM based models improve substantially, while
the ESIM-based models improve only on the more difficult MultiNLI dataset. Compared to previously published state-of-the-art systems, our models acquit themselves very well on the MultiNLI
benchmark, and competitively on the SNLI benchmark. In parallel to this work, Gong et al. (2017)
developed a novel task-specific architecture for RTE that achieves slightly better performance on
MultiNLI than our ESIM+reading+knowledge based models.8 It is worth observing that with our
knowledge-enhanced embedding architecture, our generic BiLSTM-based task model outperforms
ESIM on MultiNLI, which is architecturally much more complex and designed specifically for the
RTE task. Finally, we remark that despite careful tuning, our re-implementation of ESIM fails to
2

http://conceptnet.io/
http://wiktionary.org/
4
http://compling.hss.ntu.edu.sg/omw/
5
http://dbpedia.org/
6
We exclude ConceptNet 4 assertions created by only one contributor and from Verbosity to reduce noise.
7
Due to restrictions on code sharing, we are not able to use the public evaluation server to obtain test set
scores for SQuAD. However, for the remaining tasks we report both development accuracy and held-out test
set performance.
8
Our reading+knowledge refinement architecture can be used of course with this new model.
3

6

Model

SNLI
Dev
Test

MNLI Matched
Dev
Test

MNLI Mismatched
Dev
Test

BiLSTM
+ reading
+ reading + knowledge

84.4
86.1
86.5

83.4
85.4
85.7

70.0
75.3
76.8

69.8
75.8
77.3

70.2
76.3
77.5

69.4
74.9
76.3

ESIM
+ reading
+ reading + knowledge

88.2
88.0
87.8

87.2
87.3
87.3

76.8
77.8
78.8

76.3
77.8
78.2

77.3
78.4
78.8

75.8
77.0
77.0

−
−

88.03
88.61

−
−

74.52
78.83

−
−

73.52
77.83

SotA Results

Table 2: Results on the SNLI as well as MultiNLI-Matched and -Mismatched for n =
300−dimensional models. The model using external knowledge is trained with the top-20 retrieved
ConceptNet assertions. 1 Chen et al. (2017b), 2 Gong et al. (2017). 3 Chen et al. (2017a).
Dim/Data

1/1k

3/3k

10/10k

30/30k

100/100k

300/Full

69.7
71.0(+1.3)
73.8(+2.8)

75.5
75.0(-0.5)
77.4(+2.4)

81.3
80.6(-0.7)
81.3(+0.7)

88.2
88.0(-0.2)
87.8(-0.2)

68.1
70.7(+2.6)
71.3(+0.6)

76.9
78.1(+1.2)
78.8(+0.7)

SNLI
ESIM
+ reading
+ reading + knowledge

52.2
60.9(+8.7)
62.9(+2.0)

ESIM
+ reading
+ reading + knowledge

44.3
51.8(+7.5)
52.4(+0.6)

59.3
66.0(+6.7)
68.6(+2.6)

MultiNLI Matched + Mismatched
50.0
55.8(+5.8)
57.9(+2.1)

55.5
60.1(+4.6)
62.4(+2.3)

61.9
65.0(+3.1)
66.6(+1.6)

Table 3: Development set results when reducing training data and embedding dimsensionality with
PCA. In parenthesis we report the relative differences to the respective result directly above.

match the 88% reported in Chen et al. (2017b); however, with MultiNLI, we find that our implementation of ESIM performs considerably better (by approximately 5%). The instability of the results
suggests, as well as the failure of a custom RTE-architecture to consistently perform well suggests
that current SotA RTE models may be overfit to the SNLI dataset.
5.3

R EDUCING T RAINING DATA & D IMENSIONALITY OF P RE - TRAINED W ORD
E MBEDDINGS

We find that there is only little impact when using external knowledge on the RTE task when using a more sophisticated task model such as ESIM. We hypothesize that the attention mechanisms
within ESIM jointly with powerful, pre-trained word representations allow for the recovery of some
important lexical relations when trained on a large dataset. It follows that by reducing the number
of training data and impoverishing pre-trained word representations the impact of using external
knowledge should become larger.
To test this hypothesis, we gradually impoverish pre-trained word embeddings by reducing their
dimensionality with PCA while reducing the number of training instances at the same time.9 Our
joint data and dimensionality reduction results are presented in Table 3. They show that there is
indeed a slightly larger benefit when employing background knowledge in the more impoverished
settings with largest improvements over using only the novel reading architecture when using around
10k examples and reduced dimensionality to 10. However, we observe that the biggest overall
impact over the baseline ESIM model stems from our contextual refinement strategy (reading) which
is especially pronounced for the 1k and 3k experiments. This highlights once more the usefulness
of our refinement strategy even without the use of additional knowledge.
9
Although reducing either embedding dimensionality or data individually exhibit similar (but less pronounced) results we only report the joint reduction results here.

7

p:
h:

His off-the-cuff style seems amateurish [...]
He didn’t look like an amateur

the net cost of operations.
The gross cost.

but uh these guys [...] file their uh their
final exams [...]
These men filed their midterm exams [...]

a:
→

look like synonym seem
contradiction

gross antonym net
contradiction

midterm antonym final
contradiction

ā:
→

look like antonym seem
entailment

gross synonym net
entailment

midterm synonym final
entailment

Table 4: Three examples for the antonym ↔ synonym swapping experiment on MultiNLI. p-premise,
h-hypothesis, a-assertion, ā-swapped assertion.

(a) BiLSTM on MultiNLI.

(b) ESIM on MultiNLI.

(c) BiLSTM on SQuAD.

Figure 2: Performance differences when ignoring certain types of knowledge, i.e., relation predicates
during evaluation. Normalized performance differences are measured on the subset of examples for
which an assertion of the respective relation predicate occurs.

6

A NALYSIS OF K NOWLEDGE U TILIZATION

Is additional knowledge used? To verify whether and how our models make use of additional
knowledge, we conducted several experiments. First, we evaluated models trained with knowledge
on our tasks while not providing any knowledge at test time. This ablation drops performance by
3.7–3.9% accuracy on MultiNLI, and by 4% F1 on SQuAD. This indicates the model is refining the
representations using the provided assertions in a useful way.
Are models sensitive to semantics of the provided knowledge? The previous result does not
show that the models utilize the provided assertions in any consistent way (it may just reflect a
mismatch of training and testing conditions). Therefore, to test our models sensitivity towards the
semantics of the assertions, we run an experiment in which we swap the synonym with the antonym
predicate in the provided assertions during test time. Because of our heuristic retrieval mechanism,
not all such “counterfactuals” will affect the truth of the inference, but we still expect to see a more
significant impact. The performance drop on MultiNLI examples for which either a synonym or an
antonym-assertion is retrieved is about 10% for both the BiLSTM and the ESIM model. This very
large drop clearly shows that our models are sensitive to the semantics of the provided knowledge.
Examples of prediction changes are presented in Table 4. They demonstrate that the system has
learned to trust the presented assertions to the point that it will make appropriate counterfactual
inferences—that is, the change in knowledge has caused the change in prediction.
What knowledge is used? After establishing that our models are somehow sensitive to semantics
we wanted to find out which type of knowledge is important for which task. For this analysis
we exclude assertions including the most prominent predicates in our knowledge base individually
when evaluating our models. The results are presented in Figure 2. They demonstrate that the
biggest performance drop in total (blue bars) stems from related to assertions. This very prominent
predicate appears much more frequently than other assertions and helps connecting related parts of
the 2 input sequences with each other. We believe that related to assertions offer benefits mainly
from a modeling perspective by strongly connecting the input sequences with each other and thus
8

bridging long-range dependencies similar to attention. Looking at the relative drops obtained by
normalizing the performance differences on the actually affected examples (green) we find that our
models depend highly on the presence of antonym and synonym assertions for all tasks as well as
partially on is a and derived from assertions. This is an interesting finding which shows that the
sensitivity of our models is selective wrt. the type of knowledge and task. The fact that the largest
relative impact stems from antonyms is very interesting because it is known that such information is
hard to capture with distributional semantics contained in pre-trained word embeddings.

7

R ELATED W ORK

The role of background knowledge in natural language understanding has long been remarked on,
especially in the context of classical models of AI (Schank & Abelson, 1977; Minsky, 2000); however, it has only recently begun to play a role in neural network models of NLU (Ahn et al., 2016;
Xu et al., 2016; Long et al., 2017; Dhingra et al., 2017). However, previous efforts have focused on
specific tasks or certain kinds of knowledge, whereas we take a step towards a more general-purpose
solution for the integration of heterogeneous knowledge for NLU systems by providing a simple,
general-purpose reading architecture that can read background knowledge encoded in simple natural
language statements, e.g., “abdication is a type of resignation”.
Bahdanau et al. (2017) use textual word definitions as a source of information about the embeddings
of OOV words. In the area of visual question answering Wu et al. (2016) utilize external knowledge
in form of DBpedia comments (short abstracts/definitions) to improve the answering ability of a
model. Marino et al. (2017) explicitly incorporate knowledge graphs into an image classification
model. Xu et al. (2016) created a recall mechanism into a standard LSTM cell that retrieves pieces
of external knowledge encoded by a single representation for a conversation model. Concurrently,
Dhingra et al. (2017) exploit linguistic knowledge using MAGE-GRUs, an adapation of GRUs to
handle graphs, however, external knowledge has to be present in form of triples. The main difference to our approach is that we incorporate external knowledge in free text form on the word level
prior to processing the task at hand which constitutes a more flexible setup. Ahn et al. (2016) exploit knowledge base facts about mentioned entities for neural language models. Bahdanau et al.
(2017) and Long et al. (2017) create word embeddings on-the-fly by reading word definitions prior
to processing the task at hand. Pilehvar et al. (2017) seamlessly incorporate information about word
senses into their representations before solving the downstream NLU task, which is similar. We go
one step further by seamlessly integrating all kinds of fine-grained assertions about concepts that
might be relevant for the task at hand.
Another important aspect of our approach is the notion of dynamically updating wordrepresentations. Tracking and updating concepts, entities or sentences with dynamic memories is
a very active research direction (Kumar et al., 2016; Henaff et al., 2017; Ji et al., 2017; Kobayashi
et al., 2017). However, those works typically focus on particular tasks whereas our approach is taskagnostic and most importantly allows for the integration of external background knowledge. Other
related work includes storing temporary information in weight matrices instead of explicit neural
activations (such as word representations) as a biologically more plausible alternative.

8

C ONCLUSION

We have presented a novel task-agnostic reading architecture that allows for the dynamic integration
of background knowledge into neural NLU models. Our solution, which is based on the incremental
refinement of word representations by reading supplementary inputs, is flexible and be used with
virtually any existing NLU architecture that rely on word embeddings as input. Our results show that
embedding refinement using both the system’s text inputs, as well as supplementary texts encoding
background knowledge can yield large improvements. In particular, we have shown that relatively
simple task architectures (e.g., based on simple BiLSTM readers) can become competitive with
state-of-the-art, task-specific architectures when augmented with our reading architecture.
9

ACKNOWLEDGMENTS
This research was conducted during an internship of the first author at DeepMind. It was further
partially supported by the German Federal Ministry of Education and Research (BMBF) through
the projects ALL SIDES (01IW14002), BBDC (01IS14013E), and Software Campus (01IS12050,
sub-project GeNIE).

R EFERENCES
Sungjin Ahn, Heeyoul Choi, Tanel Pärnamaa, and Yoshua Bengio. A neural knowledge language
model. arXiv, 2016.
Dzmitry Bahdanau, Tom Bosc, Stanisław Jastrzebski, Edward Grefenstette, Pascal Vincent, and
Yoshua Bengio. Learning to compute word embeddings on the fly. arXiv, 2017.
Samuel R Bowman, Gabor Angeli, Potts Christopher, and Christopher D Manning. A large annotated corpus for learning natural language inference. In EMNLP. Association for Computational
Linguistics, 2015.
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. Recurrent neural network-based sentence encoder with gated attention for natural language inference. arXiv,
2017a.
Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, and Hui Jiang. Enhancing and Combining Sequential and Tree LSTM for Natural Language Inference. ACL, 2017b.
Bhuwan Dhingra, Zhilin Yang, William W Cohen, and Ruslan Salakhutdinov. Linguistic Knowledge
as Memory for Recurrent Neural Networks. arXiv, 2017.
Yichen Gong, Heng Luo, and Jian Zhang. Natural language inference over interaction space. arXiv,
2017.
Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. Tracking the world
state with recurrent entity networks. In ICLR, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1997.
Minghao Hu, Yuxing Peng, and Xipeng Qiu. Reinforced mnemonic reader for machine comprehension. arXiv, 2017.
Yangfeng Ji, Chenhao Tan, Sebastian Martschat, Yejin Choi, and Noah A Smith. Dynamic entity
representations in neural language models. In EMNLP, 2017.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly
supervised challenge dataset for reading comprehension. In ACL, July 2017.
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. ICLR, 2015.
Sosuke Kobayashi, Naoaki Okazaki, and Kentaro Inui. A neural language model for dynamically representing the meanings of unknown words and entities in a discourse. arXiv preprint
arXiv:1709.01679, 2017.
Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor
Zhong, Romain Paulus, and Richard Socher. Ask me anything: Dynamic memory networks for
natural language processing. In ICML, pp. 1378–1387, 2016.
Teng Long, Emmanuel Bengio, Ryan Lowe, Jackie Chi Kit Cheung, and Doina Precup. World
knowledge for reading comprehension: Rare entity prediction with hierarchical lstms using external descriptions. In EMNLP, 2017.
Christopher D Manning, Hinrich Schütze, et al. Foundations of statistical natural language processing, volume 999. MIT Press, 1999.
10

Kenneth Marino, Ruslan Salakhutdinov, and Abhinav Gupta. The more you know: Using knowledge
graphs for image classification. CVPR, 2017.
Marvin Minsky. Commonsense-based interfaces. Communications of the ACM, 2000.
Bhaskar Mitra and Nick Craswell.
arXiv:1705.01509, 2017.

Neural models for information retrieval.

arXiv preprint

Karthik Narasimhan, Adam Yala, and Regina Barzilay. Improving information extraction by acquiring external evidence with reinforcement learning. In EMNLP, 2016.
Rodrigo Nogueira and Kyunghyun Cho. Task-oriented query reformulation with reinforcement
learning. In EMNLP, 2017.
Boyuan Pan, Hao Li, Zhou Zhao, Bin Cao, Deng Cai, and Xiaofei He. Memen: Multi-layer embedding with memory networks for machine comprehension. arXiv, 2017.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word
representation. In EMNLP, 2014.
Mohammad Taher Pilehvar, Jose Camacho-Collados, Roberto Navigli, and Nigel Collier. Towards
a seamless integration of word senses into downstream nlp applications. In ACL, 2017.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. In SQuAD: 100,000+ Questions for Machine Comprehension of Text, 2016.
Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomás Kociský, and Phil Blunsom.
Reasoning about entailment with neural attention. ICLR, 2015.
Roger Schank and Robert Abelson. Scripts, Plans, Goals, and Understanding. Psychology Press,
1977.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hananneh Hajishirzi. Bi-Directional Attention
Flow for Machine Comprehension. In ICLR, 2017.
Robert Speer and Catherine Havasi. Representing General Relational Knowledge in ConceptNet 5.
In LREC, 2012.
Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. Gated self-matching networks
for reading comprehension and question answering. In ACL, 2017.
Dirk Weissenborn, Georg Wiese, and Laura Seiffe. Making Neural QA as Simple as Possible but
not Simpler. In CoNLL, 2017.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for
sentence understanding through inference. arXiv, 2017.
Qi Wu, Peng Wang, Chunhua Shen, Anton van den Hengel, and Anthony Dick. Ask Me Anything:
Free-form Visual Question Answering Based on Knowledge from External Sources. CVPR, 2016.
Zhen Xu, Bingquan Liu, Baoxun Wang, Chengjie Sun, and Xiaolong Wang. Incorporating LooseStructured Knowledge into LSTM with Recall Gate for Conversation Modeling. arXiv, 2016.

11

A

I MPLEMENTATION D ETAILS

In the following we explain the detailed implementation of our two task-specific, baseline models.
We assume to have computed the contextually (un-)refined word representations depending on the
setup and embedded our input sequences q = (q1 , ..., qLQ ) and p = (p1 , ..., pLP ) to Q ∈ Rn×LQ
and P ∈ Rn×LP , respectively. The word representation update gate in Eq. 4 is initialized with
a bias of 1 to refine representations only slightly in the beginning of training. In the following
as before, we denote the hidden dimensionality of our model by n and a fully-connected layer by
FC(z) = Wz + b, W ∈ Rn×m , b ∈ Rn , u ∈ Rm .
A.1

Q UESTION A NSWERING

Encoding In the DQA task q refers to the question and p to the supporting text. At first we process
both sequences by identical BiLSTMs in parallel (Eq. 6) followed by separate linear projections
(Eq. 7) .

Q̂ = BiLSTM(Q)

P̂ = BiLSTM(P)

Q̃ = UQ Q̂

Q̂ ∈ R2n×LQ , P̂ ∈ R2n×LP
Q̃ ∈ R

P̃ = UP P̂

n×LQ

, P̃ ∈ R

n×LP

,

(6)
(7)

UQ , UP ∈ Rn×2n are initialized by [I; I] where I ∈ Rn×n is the identity matrix.
Prediction Our prediction– or answer layer is the same as in Weissenborn et al. (2017). We first
compute a weighted, n-dimensional representation q̃ of the processed question Q̃ (Eq. 8).
α = softmax(vq Q̃) , vq ∈ Rn
X
q̃ =
αi q̃i

(8)

i

The probability distribution ps /pe for the start/end location of the answer is computed by a 2-layer
MLP with a ReLU activated, hidden layer sj as follows:
"
sj = ReLU FCs

p̃j
q̃
p̃j

ps (j) ∝ exp(vs sj )

#!!

"
ej = ReLU FCe

q̃

vs ∈ R

p̃j
q̃

p̃j

n

pe (j) ∝ exp(ve sj )

#!!
q̃

v e ∈ Rn

(9)

The model is trained to minimize the cross-entropy loss of the predicted start and end positions,
respectively. During evaluation we extract the span (i, k) with the best span-score ps (i) · pe (k) of
maximum token length k − i = 16.
A.2

R ECOGNIZING T EXTUAL E NTAILMENT

Encoding Analogous to DQA we encode our input sequences by BiLSTMs, however, for RTE
we use conditional encoding (Rocktäschel et al., 2015) instead. Therefore, we initially process the
embedded hypothesis Q by a BiLSTM and use the respective end states of the forward and backward
LSTM as initial states for the forward and backward LSTM that processes the embedded premise
P.
Prediction We hconcatenate
i the outputs of the forward and backward LSTMs processing the
fw
bw
premise p, i.e., p̃t ; p̃t
∈ R2n and run each of the resulting LP outputs through a fullyconnected layer with ReLU activation (ht ) followed by a max-pooling operation over time resulting
in a hidden state h ∈ Rn . Finally, h is used to predict the RTE label as follows:
12


 f w 
p̃t
ht = ReLU FC
p̃bw
t
h = maxpool ht
t

p(c) ∝ exp(vc h) , vc ∈ Rn

(10)

The probability of choosing category c ∈ {entailment, contradiction, neutral} is defined in Eq. 10.
Finally, the model is trained to minimize the cross-entropy loss of the predicted category probability
distribution p.
A.3

T RAINING

As pre-processing steps we lowercase all inputs and tokenize it. Additionally, we make use of
lemmatization as described §3.2 in which is necessary for matching. As pre-trained word representations we use 300-dimensional word-embeddings from Glove (Pennington et al., 2014). We
employed ADAM (Kingma & Ba, 2015) for optimization with an initial learning-rate of 10−3 which
was halved whenever the F1 measure (DQA) or the accuracy (RTE) dropped on the development
set between 1000/2000 minibatches for DQA and RTE respectively. We used mini-batches of size
16 for DQA and 64 for RTE. Additionally, for regularization we make use of dropout with a rate of
0.2 on the computed non-contextual word representations ew defined in §3.1 with the same dropout
mask for all words in a batch. All our models were trained with 3 different random seeds and the
top performance is reported.

13

