Type-level word embeddings use the same set of parameters to represent all instances of a word regardless of its context, ignoring the inherent lexical ambiguity in language. Instead, we embed semantic concepts (or synsets) as defined in WordNet and represent a word token in a particular context by estimating a distribution over relevant semantic concepts. We use the new, context-sensitive embeddings in a model for predicting prepositional phrase(PP) attachments and jointly learn the concept embeddings and model parameters. We show that using context-sensitive embeddings improves the accuracy of the PP attachment model by 5.4 absolute points, which amounts to a 34.4 relative reduction in errors.
@cite_5: The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible.
There is a large body of work that tried to improve word embeddings using external resources. extended the CBOW model @cite_5 by adding an extra term in the training objective for generating words conditioned on similar words according to a lexicon. extended the skipgram model @cite_5 by representing word senses as latent variables in the generation process, and used a structured prior based on the ontology. used belief propagation to update pre-trained word embeddings on a graph that encodes lexical relationships in the ontology. Similarly, improved word embeddings by representing each sense of the word in a way that reflects the topology of the semantic network they belong to, and then representing the words as convex combinations of their senses. In contrast to previous work that was aimed at improving word representations, we propose an approach for obtaining embeddings at the , while jointly optimizing the model parameters for the NLP task of interest.