Neural language models are a powerful tool to embed words into semantic vector spaces. However, learning such models generally relies on the availability of abundant and diverse training examples. In highly specialised domains this requirement may not be met due to difficulties in obtaining a large corpus, or the limited range of expression in average use. Such domains may encode prior knowledge about entities in a knowledge base or ontology. We propose a generative model which integrates evidence from diverse data sources, enabling the sharing of semantic information. We achieve this by generalising the concept of co-occurrence from distributional semantics to include other relationships between entities or words, which we model as affine transformations on the embedding space. We demonstrate the effectiveness of this approach by outperforming recent models on a link prediction task and demonstrating its ability to profit from partially or fully unobserved data training labels. We further demonstrate the usefulness of learning from different data sources with overlapping vocabularies.
@cite_4: The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible.
Relational data can also be used to learn distributed representations of entities in knowledge graphs, entities which may correspond to or can be mapped to words. A general approach is to implicitly embed the graph structure through vertex embeddings and rules (or transformations) for traversing it. bordes2011learning scored the similarity of entities under a given relationship by their distance after transformation using pairs of relationship-specific matrices. socher2013reasoning describe a neural network architecture with a more complex scoring function, noting that the previous method does not allow for interactions between entities. The TransE model of bordes2013translating (and extensions such as wang2014knowledge , fan2014transition , and lin2015learning ) represents relationships as , motivated by the tree representation of hierarchical relationships, and observations that linear composition of entities appears to preserve semantic meaning @cite_4 . These approaches are uniquely concerned with relational data however, and do not consider distributional semantics from free text. faruqui2015retrofitting and johansson-nietopina:2015:NAACL-HLT describe methods to modify pre-existing word embeddings to align them with evidence derived from a knowledge base, although their models do not learn representations .