Incorporating Literals into Knowledge Graph Embeddings

Abstract

Knowledge graphs (KGs) form the backbone of a range of
applications, for instance in the areas of search, question answering and data integration. Some well known knowledge
graphs are DBpedia [Lehmann et al., 2015], Freebase [Bollacker et al., 2008], YAGO3 [Mahdisoltani et al., 2014] and
the Google Knowledge Graph. There are different knowledge
representation paradigms for modeling knowledge graphs
such as the Resource Description Framework (RDF) and (labeled) property graphs. Within this paper, we consider a
knowledge graph to be a set of triples, where each triple connects an entity (shown as circle in Figure 1) to another entity
or a literal (the latter shown as rectangle in Figure 1) via relationships. Such knowledge graphs can be represented by the
∗

Equal contribution

di

es

At

di

es
A
t

stu

ar

Jane

2001

birthY
e

r

Introduction

st
u

friendOf ?

John

a
Ye

1

Doe High
School

th

Knowledge graphs, on top of entities and their relationships, contain another important element: literals. Literals encode interesting properties (e.g. the
height) of entities that are not captured by links between entities alone. Most of the existing work
on embedding (or latent feature) based knowledge
graph modeling focuses mainly on the relations
between entities. In this work, we study the effect of incorporating literal information into existing knowledge graph models. Our approach,
which we name LiteralE, is an extension that can be
plugged into existing latent feature methods. LiteralE merges entity embeddings with their literal
information using a learnable, parametrized function, such as a simple linear or nonlinear transformation, or a multilayer neural network. We extend
several popular embedding models using LiteralE
and evaluate the performance on the task of link
prediction. Despite its simplicity, LiteralE proves
to be an effective way to incorporate literal information into existing embedding based models,
improving their performance on different standard
datasets, which we augmented with their literals
and provide as testbed for further research.

bir

arXiv:1802.00934v1 [] 3 Feb 2018

Agustinus Kristiadi∗1 , Mohammad Asif Khan∗1 , Denis Lukovnikov1 , Jens Lehmann1,2 , Asja Fischer1
1
SDA Group, University of Bonn
2
EIS Department, Fraunhofer IAIS
{kristiadi, s6mokhan}@uni-bonn.de, lukovnik@cs.uni-bonn.de,
jens.lehmann@uni-bonn.de, jens.lehmann@iais.fraunhofer.de, asja.fischer@gmail.com

2000

Figure 1: Literals encode information that cannot be represented by
relations alone, and are useful for link prediction task. For instance,
by considering both birthYear literals and the fact that John and
Jane both study at Doe High School, we can infer the relation
friendOf between John and Jane is likely to exist.

RDF as well as the property graph paradigms, i.e. the methods presented in this paper are applicable to both.
Knowledge graphs aim to capture factual knowledge
within a particular domain. However, they are often incomplete since, e.g., more information is provided for popular
entities or because the knowledge graph is partially or fully
generated via an automatic extraction process. As a result,
knowledge graphs rely heavily on methods predicting unknown triples given all known triples. This problem is usually
referred to as link prediction. The closely related problem of
detecting incorrect triples in knowledge graphs is referred to
as link correction and relevant for improving the quality of
knowledge graphs.
Due to the importance of the problem, many methods for
link prediction and correction in knowledge graphs have been
developed. The two main classes of these methods are graph
feature and latent feature models. Graph feature models predict the existence of triples based on features directly ob-

served in the knowledge graph, such as the neighborhood of
an entity and paths to other entities. They are well suited for
modeling local graph patterns. In latent feature models, lowdimensional, latent representations (also called embeddings)
of entities and relations are learned. These embeddings incorporate the structure of a knowledge graph, can capture global
patterns and allow to compute the likeliness of a given triple
in terms of a probability or score function.
However, most of the recent work on latent feature models
only take entities and their relations to other entities into account. Therefore, they are potentially missing the additional
information encoded in literals. For example, Figure 1 shows
two entities with both structural (visiting the same school) as
well as literal (birth years) information. To maximize the accuracy of predicting a friendOf relation between entities,
structural and literal information should be combined as people visiting the same school and with similar age tend to have
a higher probability of being friends.
In this paper, we investigate the advantage obtained by incorporating the additional information provided by literals
into latent feature models. We introduce LiteralE, a method to
enrich entity embeddigs with their literal information. Given
an entity embedding, we incorporate its corresponding literals using a learnable parametric function, e.g. an affine transformation or a neural network. The resulting literal-enriched
embeddings can then replace the vanilla embeddings in any
latent feature model. LiteralE can be seen as an extension
module that can be universally combined with the score function of any existing latent feature model. The resulting system can be jointly trained with stochastic gradient descent in
an end-to-end manner. Thus, LiteralE is a simple approach to
incorporate literal information into existing models. Within
this paper, we focus on numerical literal values. However,
the principle can be directly generalized to other literal types,
such as textual and image information. Our contributions in
this paper are threefold:
• We introduce LiteralE, a universal approach to enrich latent feature models with literal information via a learnable parametric function. In contrast to other latent feature models including literals, our approach does not
require hand-crafted features, does not rely on a fixed
function to combine entity embeddings and literals, can
model interactions between an embedding of an entity
and all its literal values and can be trained end-to-end.
• We evaluate LiteralE on standard link prediction datasets
(FB15k, FB15k-237 and YAGO3-10). We extended
FB15k and FB15k-237 with literals, in order to allow
for direct comparison against other methods on these
standard datasets. We provide these literal-extended versions and hope they can serve as a test bed for future
research on literals in KG modeling. 1
• Based on experimental results on the extended datasets,
we show that by exploiting the information provided by
literals, the link prediction performance of existing models can be increased significantly.
1

A literal-extended version of YAGO3-10 is provided by
[Pezeshkpour et al., 2017]

Our implementation and all datasets are publicly available
at: https://github.com/SmartDataAnalytics/
LiteralE.

2

Preliminaries

Problem Description Link prediction is defined as the task
of deciding whether a fact (represented by a triple) is true
or false given the knowledge graph. More formally, let
E = {e1 , · · · , eNe } be the set of entities, L be the set of all literal values, R = {r1 , · · · , rNr } the set of relations connecting two entities and D = {d1 , · · · , dNd } the set of relations
connecting an entity and a literal, i.e., the data relations. A
knowledge graph G is a subset of (E × R × E) ∪ (E × D × L)
representing the facts that are assumed to hold. Then, link
prediction can be formulated as the the task of mapping each
possible triple (ei , rk , ej ) ∈ E ×R×E to a score value, where
a higher value implies the triple is more likely to be true.
Latent Feature Models In general, latent feature models are a class of methods in which low dimensional vector representations of entities and relations, called embeddings, are learned. Let H be the dimension of latent variables. Let E ∈ RNe ×H be the entity embeddings matrix and
R ∈ RNr ×H be the relation embeddings matrix for E and
R respectively. The task of link prediction is then finding
a function f : RH × RH × RH → R that maps a triple’s
embeddings (ei , ej , rk ) to a score f (ei , ej , rk ) that correlates
with the truth value of the triple.
Latent feature models for link predictions are well studied. Examples of popular methods are DistMult [Yang
et al., 2014], ComplEx [Trouillon et al., 2016], and
ConvE [Dettmers et al., 2017]. These methods follow a
score-based approach and differ in their parameterization of
the entity and relation representations. Some use latent matrices instead of latent vectors or allow to use relation and entity
embeddings to have different dimensions. The exact formulation of the scoring function f that uses these embeddings
also differs.
The DistMult scoring function is defined as diagonal bilinear interaction between two entity and a relation embedding
in a given triple, as follows:

fDistMult (ei , ej , rk ) = hei , ej , rk i =

H
X

ei

ej

rk , (1)

h=1

where ei and ei denote the i-th and j-th row of E respectively,
and rk denotes the k-th row of R.
ComplEx can be seen as DistMult analogue in the complex
space. The embedding matrices have two parts: the real part
Re(E) and Re(R), and the imaginary part Im(E) and Im(R).
The scoring function is defined as
fComplEx (ei , ej , rk ) = hRe(ei ), Re(ej ), Re(rk )i
+ hIm(ei ), Im(ej ), Re(rk )i
+ hRe(ei ), Im(ej ), Im(rk )i
− hIm(ei ), Re(ej ), Im(rk )i .

(2)

ComplEx thus has twice the number of parameters compared to DistMult but provides the benefit of modelling
asymmetric relationships better, as discussed by Trouillon et
al. [2016].
ConvE uses convolution operations to extract features from
entity and relation embeddings. Let h be a nonlinear function,
ω be convolution filters, and W be a weight matrix. The
ConvE score function is then defined as follows:
fConvE (ei , ej , rk ) = h(vec(h([ei , rk ] ∗ ω))W)eo .

Triple

LiteralE

gMLP (ei , li ) = h(W|2 h(W|1 [ei , li ]) ,

Score Func.

Score

f

y

g

ei

rk

(3)

Our method of incorporating literals into existing latent feature models, which we call LiteralE, is a simple, modular,
and universal extension which can enhance the performance
of the base models.
Let L ∈ RNe ×Nd be a matrix, where each entry Lik contains the k-th literal value of i-th entity if a triple with the
i-th entity and the k-th data relation exists in the knowledge
graph and zero otherwise. At the core of LiteralE is a function g : RH × RNd → RH that takes entity embeddings
and their literal vectors as inputs and maps them onto another
H-dimensional vector. The resulting vectors are (non-)linear
transformations of both the entity embeddings and their literal vectors. Then, the original embedding vectors in the
scoring function of any latent feature model can be replaced
with these literal-enriched vectors. For example in our experiments, we replace every entity embedding ei with elit
i =
g(ei , li ) in the scoring functions of DistMult and ConvE. For
ComplEx, where the embeddings have a real and an imaginary part, we replace Re(ei ) and Im(ei ) with Re(ei )lit =
gRe (Re(ei ), li ) and Im(ei )lit = gIm (Im(ei ), li ), respectively.
Aside of these changes in entities embeddings, the score functions are as described before in Equations (1), (2), and (3).
The transformation function g can be any fixed or learnable
function. In this work, we focus on simple learnable functions: linear transformation, non-linear transformation and a
multi-layer neural network (MLP), as specified in the following:
glin (ei , li ) = W| [ei , li ] ,
gnonlin (ei , li ) = h(W| [ei , li ]) ,

LiteralE
li

In this work, we use DistMult, ComplEx, and ConvE as
main models which we extend by LiteralE. Our hypothesis is
that by incorporating LiteralE and thus literals information,
we can significantly improve the performance of DistMult,
ComplEx, and ConvE.

3

Embeddings

(4)
(5)
(6)

where W, W1 and W2 are trainable weight matrices and h
is a non-linear function. More concretely, we concatenate
the entity embedding vector with the corresponding literals
vector and map it into an H-dimensional vector. In Eq. 4,
the mapping is a simple linear combination with a learnable
weight matrix, whereas in Eq. 5 we add nonliniearity on top
of the linear combination. Finally, in Eq. 6, we use function
composition for this mapping, i.e. MLP. We experiment with

ej

g
lj

Figure 2: Overview on how LiteralE is applied to the base scoring
function f . LiteralE takes the embedding and the corresponding
literals as input, and combines them via a parametric function g.
The output is then being used to replace the original embedding in
the score function f .

hyperbolic tangent and ReLU for the non-linear transformation (or activation function) and only ReLU for MLP.
In summary, we propose to replace the score function
fX (ei , ej , rk ) from the host method X with the function composition fX (g(ei , li ), g(ej , lj ), rk ) as illustrated in Figure 2.
This new scoring function can be trained by gradient descent
using the same training procedure as before.

4

Related Work

In the last years, several efforts to incorporate literals
into knowledge graph embedding models have been made.
Toutantova et al. [2015] and Tu et al. [2017] make use of
textual literals of entities in addition to relational embeddings. More specifically they learn additional entity embeddings from their textual description and use them in an additive term in the score function of latent distant models. Xie
et al., [2016] use image literals in their model by projecting entities’ image features into an entity embeddings space.
However all of those approaches do not consider numerical
literals. MultiModal [Pezeshkpour et al., 2017] extends DistMult to also predict the likeliness of (subject, predicate, literal) triples, by replacing the object embedding in standard
DistMult by its literal embedding (where literals of different modalities are taken into account). By doing so literals
are incorporated into entity embedding by multi-task learning scheme. In contrast, LiteralE combines the literals into
the entity embedding directly and thus no additional task is
required.
KBLRN [Garcia-Duran and Niepert, 2017] handles literals
in a separate function added to the vanilla scoring function
and thus does not incorporate literals in to the entity embeddings themselves2 Moreover, KBLRN’s numerical features
2

Note, that incorporating the literal information into the embeddings also seems advantageous for entity disambiguation or clustering.

Dataset
# Entities (Ne )
# Relations (Nr )
# Data rel. (Nd )
# Literals (|L|)
# Relational triples
# Literal triples

FB15k

FB15k-237

YAGO3-10

14,951
1,345
121
18,741
592,213
70,257

14,541
237
121
18,741
310,116
70,257

123,182
37
5
111,406
1,089,040
111,406

Table 1: Statistics of number of entities, relations, and literals for all
datasets used in this paper.

are handcrafted, which requires prior knowledge and manual intervention. In contrast, LiteralE learns to merge entity
embeddings and its literals in a single representation in an
end-to-end manner.
MTKGNN [Tay et al., 2017] extends ERMLP [Dong et al.,
2014] and incorporates literals by introducing an additional
learning task (i.e. predicting the literal value for a given entity). The multi-task learning approach of MTKGNN requires
an additional attribute-specific training procedure. Scalability when adding another type or modality of literals is thus
costly as another learning task needs to be devised.
Lastly, the model recently proposed by Thoma et al. [2017]
can be seen as a special case of LiteralE where the function
used to combine literals of entities is a concatenation followed by singular value decomposition. Thus, they use fixed
functions to combine the representations, whereas LiteralE
employs adaptable functions and is therefore more flexible.
Furthermore, they only consider image and text literals and
thus do not consider numerical literals.

5
5.1

Experiments
Training

In our experiments, we use the same training approach as
Dettmers et al. [2017] for all the tested models. That is,
for every given triple (ei , rk , ej ), we compute scores for
(ei , rk , e0j ), ∀e0j ∈ E using the (original or LiteralE-extended)
scoring function f , and use the sigmoid function to produce
probabilities p = σ(f (·)). The model is trained by minimizing the binary cross-entropy loss of the produced probability vector p ∈ [0, 1]Ne w.r.t. the vector of truth values
y ∈ {0, 1}Ne for the scored triples:
L(p, y) = −

Ne
1 X
(y log(px )+(1−yx ) log(1−px )), (7)
Ne x=1 x

where px and yx are the predicted probability and the
given truth value for the x-th element of our candidate set
{(ei , rk , e0j ), ∀e0j ∈ E}. We use Adam [Kingma and Ba,
2014] to optimize this loss function.

5.2

Datasets

We use three widely used datasets for evaluating link prediction performance: FB15k, FB15k-237, and YAGO3-10.
FB15k [Bordes et al., 2013] is a subset of Freebase where
most triples are related to movies and sports. As discussed
by Dettmers et al., [2017] FB15k has a large number of test

triples which can simply be obtained by inverting training
triples. This results in a biased test set, for which a simple
model which is symmetric with respect to object and subject
entity is capable of achieving excellent results. To address
this problem, Toutanova and Chen [2015] created FB15k237 by removing inverse relations from FB15k. YAGO310 [Mahdisoltani et al., 2014] is a subset of the YAGO3
knowledge graph which mostly consists of triples related to
people.
In this work, we only consider numerical literals, e.g. longitude, latitude, population, age, date of birth, etc. To enrich
FB15k and FB15k-237 with literals, we created a SPARQL
endpoint for Freebase and extracted literals of all entities contained in FB15k. We further filtered the extracted literals
based on their frequency, i.e we only consider data relations
rk ∈ D that occur at least in 5 triples in FB15k. We also remove all key and ID relations since their values are not meaningful as quantities. For YAGO3-10, we use numerical literals provided by YAGO3-10-plus [Pezeshkpour et al., 2017],
which is publicly available 3 . In case an entity has multiple
literal values for a particular data relation, we arbitrarily select one of them. Some statistics of the datasets are provided
in Table 1.
Our literal-extended versions of FB15k and FB15k237 are available at:
https://github.com/
SmartDataAnalytics/LiteralE.

5.3

Experimental Setup

We build our LiteralE code on top of ConvE’s codebase,
which is publicly available4 . The hyperparameters used in all
of our experiments across all datasets are: learning rate 0.001,
batch size 128, embedding size 100, embedding dropout
probability 0.2, feature map dropout probability 0.2, projection layer dropout probability 0.3, and label smoothing 0.1.
The hyperparameter values are the same as in the experiments
of Dettmers et al. [2017] except for embedding dimensions
for DistMult and ComplEx, which we set to 100 instead of
200 in all our experiments5 .
Except for ConvE, we run all of our experiments for a maximum of 100 epochs as we observed that this is sufficient for
convergence in most cases. For ConvE, we used at most 1000
epochs, as described in the original paper [Dettmers et al.,
2017]. We apply early stopping in all of the experiments by
monitoring the Mean Reciprocal Rank (MRR) metric on the
validation set every three epochs.
To validate our approach and to eliminate the effect of
different environment setups, we reproduce the base models
and compare them with their LiteralE-extension. Note that
we were unable to reproduce the results reported by GarciaDuran and Niepert [2017] and Pezeshkpour et al. [2017] as
the code is not open-source at the time of this writing. We do
not compare LiteralE to the work of Tay et al. [2017] since
their results were obtained on an unpublished dataset6 and we
3

https://github.com/pouyapez/multim-kb-embeddings
https://github.com/TimDettmers/ConvE
5
We also tried with embedding size of 200 but it had little to no
impact on the results.
6
We could not obtain their dataset and thus unable to run our
4

Models

MR

MRR

Hits@1

Hits@3

Hits@10

DistMult [Dettmers et al., 2017]
ComplEx [Dettmers et al., 2017]
ConvE [Dettmers et al., 2017]
KBL (DistMult) [Garcia-Duran and Niepert, 2017]
KBLN [Garcia-Duran and Niepert, 2017]

97
64
66
69

0.654
0.692
0.745
0.774
0.783

0.546
0.599
0.67
0.712
0.726

0.733
0.759
0.801
-

0.824
0.84
0.873
0.876
0.878

DistMult (Ours)
ComplEx (Ours)
ConvE (Ours)

108
127
49

0.671
0.695
0.692

0.589
0.618
0.596

0.723
0.744
0.76

0.818
0.833
0.853

DistMult+LiteralE
ComplEx+LiteralE
ConvE+LiteralE

85
92
55

0.583
0.765
0.66

0.476
0.705
0.556

0.652
0.806
0.733

0.771
0.871
0.836

DistMult+LiteralE-Tanh
DistMult+LiteralE-ReLU
DistMult+LiteralE-MLP

94
120
188

0.557
0.509
0.327

0.448
0.408
0.238

0.626
0.566
0.36

0.748
0.697
0.496

Table 2: Link prediction results on FB15k.

Models

MR

MRR

Hits@1

Hits@3

Hits@10

DistMult [Dettmers et al., 2017]
ComplEx [Dettmers et al., 2017]
ConvE [Dettmers et al., 2017]
KBL (DistMult) [Garcia-Duran and Niepert, 2017]
KBLN [Garcia-Duran and Niepert, 2017]

254
248
330
231
229

0.241
0.24
0.301
0.301
0.304

0.155
0.152
0.22
0.214
0.22

0.263
0.263
0.33
-

0.419
0.419
0.458
0.475
0.47

DistMult (Ours)
ComplEx (Ours)
ConvE (Ours)

641
652
297

0.28
0.29
0.313

0.2
0.212
0.228

0.307
0.317
0.344

0.438
0.445
0.479

DistMult+LiteralE
ComplEx+LiteralE
ConvE+LiteralE

346
441
264

0.31
0.297
0.314

0.225
0.215
0.228

0.34
0.327
0.346

0.48
0.458
0.483

DistMult+LiteralE-Tanh
DistMult+LiteralE-ReLU
DistMult+LiteralE-MLP

300
454
262

0.321
0.302
0.289

0.234
0.222
0.212

0.355
0.331
0.313

0.494
0.464
0.442

Table 3: Link prediction results on FB15k-237.

Models
[
DistMult Dettmers et al., 2017]
ComplEx [Dettmers et al., 2017]
ConvE [Dettmers et al., 2017]
Multimodal S+N [Pezeshkpour et al., 2017]

MR

MRR

Hits@1

Hits@3

Hits@10

5926
6351
2792
-

0.337
0.355
0.523
0.325

0.237
0.258
0.448
0.213

0.379
0.399
0.564
0.382

0.54
0.547
0.658
0.517

DistMult (Ours)
ComplEx (Ours)
ConvE (Ours)

2943
3768
2141

0.466
0.493
0.505

0.377
0.411
0.42

0.514
0.536
0.554

0.653
0.649
0.66

DistMult+LiteralE
ComplEx+LiteralE
ConvE+LiteralE

2705
3072
1720

0.504
0.509
0.506

0.422
0.433
0.422

0.551
0.552
0.556

0.653
0.653
0.664

DistMult+LiteralE-Tanh
DistMult+LiteralE-ReLU
DistMult+LiteralE-MLP

1685
6839
3091

0.417
0.427
0.134

0.322
0.343
0.085

0.464
0.473
0.138

0.595
0.582
0.226

Table 4: Link prediction results on YAGO3-10.

experiment on their data.

cannot reproduce their approach on our datasets because their
code is not publicly available.
We use the following standard evaluation metrics for
link prediction: Mean Rank (MR), Mean Reciprocal Rank
(MRR), Hits@1, Hits@3, and Hits@10 over the test set of
each dataset.

6

Results

The results for our experiments on link prediction are summarized in Tables 2, 3, and 4. In general, we observe that incorporating literals using LiteralE results in an improvement
over base models.
ComplEx with LiteralE results in improved performance
across all datasets. Adding LiteralE to DistMult improves
its performance on FB15k-237 and YAGO3-10 but deteriorates the performance on FB15k. Specifically, while ComplEx+LiteralE achieves 10% relative improvement on MRR
on FB15k compared to basic ComplEx, DistMult+LiteralE
drops 13% relative to basic DistMult on FB15k.
We hypothesize that this drop in performance of DistMult+LiteralE on FB15k happens due to the lower expressive power of DistMult, in particular its inability to capture
asymmetry in relations. The score function of DistMult is
symmetric, i.e. f (h, r, t) = f (t, r, h), meaning it will assign
exactly the same score to a triple if its head and tail were
swapped. Since DistMult already has difficulties in modeling
asymmetric relations on FB15k, adding literals might only
introduce noise, resulting in more lower performance. ComplEx (the complex equivalent of DistMult) solves the symmetric score problem by using the complex conjugate of the
entity embedding when used as the tail in a triple (and its normal embedding when being the head). This gives ComplEx
more expressive power, which could allow it to more easily
exploit the addition of literals to improve its predictions. Our
hypothesis that DistMult+LiteralE degrades due to symmetric
relations also seems to be confirmed by the fact that adding
LiteralE improves DistMult on FB15k-237, where all symmetric relations are discarded.
On YAGO3-10, we observed noticeably better results for
our reproductions of basic DistMult and ComplEx, while also
observing slightly worse results on basic ConvE, compared
to those reported by Dettmers et al.[2017]. We believe this
is due to Dettmers et al. [2017] using a different training approach for ConvE compared to the other methods. Specifically, they use a 1-1 approach for DistMult and ComplEx,
while using a 1-N approach for ConvE. We refer our readers
to [Dettmers et al., 2017] for more details regarding this.
We observed that LiteralE improves all the base models on
YAGO3-10, although by a smaller margin than on the other
datasets. This could be attributed to the lower number of literal relations and the relatively lower average literals per entity in YAGO3-10, compared to FB15k(-237): there are 5
literal relations and 0.9 literals per entity in YAGO3-10 compared to 121 literal relations and around 1.2 literals per entity
in both FB15k and FB15k-237. A less pronounced improvement when fewer literal relations are available is consistent
with the hypothesis that literals are useful for link prediction.

Models

MRR

Hits@1

Hits@10

DistMult
ComplEx
ConvE

0.308±0.003
0.296±0.001
0.313±0.002

0.222±0.004
0.213±0.002
0.227±0.002

0.48±0.002
0.459±0.002
0.481±0.004

Table 5: Mean and standard deviation of the results of various methods with LiteralE on FB15k-237. The samples used for calculating
these statistics are obtained by performing five runs with different
random seeds.

6.1

Functional Form of LiteralE

We show the effect of the choice of the functional form of
g in Table 2, 3, and 4. In particular, we experimented with
DistMult for the three choices of functional form of g: linear,
nonlinear, and MLP (Equations (4), (5) and (6), respectively,
see Section 3).
Based on our experiments, we observed that the best results
are achieved by defining LiteralE as a simple linear function.
Adding nonlinearities (ReLU or hyperbolic tangent) generally renders the results worse on all evaluation metrics, except
on FB15k-237, where using a hyperbolic tangent yields the
best result. Using an MLP based transformation for LiteralE
yields negative results, which might be attributed to overfitting.

6.2

Effect of Random Seeds

To estimate the uncertainty in the results we obtained, we ran
LiteralE five times on FB15k-237 with all three base models.
In each run, we picked a different random seed. In Table 5, we
report the mean and standard deviation across different runs
for different evaluation metrics. The low standard deviation
of the metrics across different runs indicate that the obtained
results are stable and thus reproducible.

7

Conclusion and Future Work

In this paper, we introduced LiteralE: a simple method to
incorporate literals in latent feature models for knowledge
graphs. LiteralE can be seen as an extension module for any
latent feature model. We showed that by applying LiteralE
to existing models such as DistMult, ComplEx, and ConvE,
their link prediction performance can be improved. Furthermore, we showed that the simplest linear form of LiteralE
performed best across most of the datasets. For future work,
LiteralE could be extended to accommodate different modalities of literals, e.g. text, images etc. Furthermore, LiteralE
could also be incorporated in other tasks besides link prediction, such as entity resolution and knowledge graph clustering.

References
[Bollacker et al., 2008] Kurt Bollacker, Colin Evans,
Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase:
a collaboratively created graph database for structuring
human knowledge. In Proceedings of the 2008 ACM
SIGMOD international conference on Management of
data, pages 1247–1250. AcM, 2008.

[Bordes et al., 2013] Antoine Bordes, Nicolas Usunier,
Alberto Garcia-Duran, Jason Weston, and Oksana
Yakhnenko.
Translating embeddings for modeling
multi-relational data. In Advances in neural information
processing systems, pages 2787–2795, 2013.
[Dettmers et al., 2017] Tim Dettmers, Pasquale Minervini,
Pontus Stenetorp, and Sebastian Riedel.
Convolutional 2d knowledge graph embeddings. arXiv preprint
arXiv:1707.01476, 2017.
[Dong et al., 2014] Xin Dong, Evgeniy Gabrilovich,
Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy,
Thomas Strohmann, Shaohua Sun, and Wei Zhang.
Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In Proceedings of the 20th
ACM SIGKDD international conference on Knowledge
discovery and data mining, pages 601–610. ACM, 2014.
[Garcia-Duran and Niepert, 2017] Alberto
Garcia-Duran
and Mathias Niepert. KBLRN: End-to-end learning of
knowledge base representations with latent, relational,
and numerical features. arXiv preprint arXiv:1709.04676,
2017.
[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba.
Adam: A method for stochastic optimization. arXiv
preprint arXiv:1412.6980, 2014.
[Lehmann et al., 2015] Jens Lehmann, Robert Isele, Max
Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N
Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick
Van Kleef, Sören Auer, et al. DBpedia–a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6(2):167–195, 2015.
[Mahdisoltani et al., 2014] Farzaneh Mahdisoltani, Joanna
Biega, and Fabian Suchanek. Yago3: A knowledge base
from multilingual wikipedias. In 7th Biennial Conference
on Innovative Data Systems Research. CIDR Conference,
2014.
[Pezeshkpour et al., 2017] Pouya Pezeshkpour, CA Irvine,
Liyan Chen, and Sameer Singh. Embedding multimodal
relational data. 2017.
[Tay et al., 2017] Yi Tay, Luu Anh Tuan, Minh C Phan, and
Siu Cheung Hui. Multi-task neural network for nondiscrete attribute prediction in knowledge graphs. In Proceedings of the 2017 ACM on Conference on Information
and Knowledge Management, pages 1029–1038. ACM,
2017.
[Thoma et al., 2017] Steffen Thoma, Achim Rettinger, and
Fabian Both. Towards holistic concept representations:
Embedding relational knowledge, visual attributes, and
distributional word semantics. In International Semantic
Web Conference, pages 694–710. Springer, 2017.
[Toutanova and Chen, 2015] Kristina Toutanova and Danqi
Chen. Observed versus latent features for knowledge base
and text inference. In Proceedings of the 3rd Workshop on
Continuous Vector Space Models and their Compositionality, pages 57–66, 2015.

[Toutanova et al., 2015] Kristina Toutanova, Danqi Chen,
Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and
Michael Gamon. Representing text for joint embedding of
text and knowledge bases. In EMNLP, volume 15, pages
1499–1509, 2015.
[Trouillon et al., 2016] Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard.
Complex embeddings for simple link prediction. In International Conference on Machine Learning, pages 2071–
2080, 2016.
[Tu et al., 2017] Cunchao Tu, Han Liu, Zhiyuan Liu, and
Maosong Sun. Cane: Context-aware network embedding
for relation modeling. In Proceedings of the 55th Annual
Meeting of the Association for Computational Linguistics
(Volume 1: Long Papers), volume 1, pages 1722–1731,
2017.
[Xie et al., 2016] Ruobing Xie, Zhiyuan Liu, Tat-seng Chua,
Huanbo Luan, and Maosong Sun. Image-embodied
knowledge representation learning.
arXiv preprint
arXiv:1609.07028, 2016.
[Yang et al., 2014] Bishan Yang, Wen-tau Yih, Xiaodong
He, Jianfeng Gao, and Li Deng. Embedding entities and
relations for learning and inference in knowledge bases.
arXiv preprint arXiv:1412.6575, 2014.

