Advances in Pre-Training Distributed Word Representations
Tomas Mikolov Edouard Grave Piotr Bojanowski Christian Puhrsch Armand Joulin
cs.CL






§ INTRODUCTION


Pre-trained continuous word representations have become
basic building blocks of many Natural Language Processing (NLP) and
Machine Learning applications. 



These pre-trained representations provide distributional information 
about words,
that typically improve the generalization of models learned on limited amount of data <cit.>.
This information is typically derived from statistics
gathered from large unlabeled corpus of text data <cit.>.
A critical aspect of their training is thus to capture efficiently as much
statistical information as possible from rich and vast sources of data.

A standard approach for learning word representations is 
to train log-bilinear models based on either the 
skip-gram or the continuous bag-of-words (cbow) architectures, as
implemented in word2vec <cit.> and fastText <cit.>[<https://fasttext.cc/>].
In the skip-gram model, nearby words are predicted given a source word, while in the cbow model,
the source word is predicted according to its context.
These architectures and their implementation
have been optimized to produce high quality word 
representations able to transfer to many tasks,
while maintaining a sufficiently high training speed to scale to massive amount of data.










Recently, word2vec representations have been widely used 
in NLP pipelines to improve their performance.
Their impressive capability at transfering to new problems 
suggests that they are capturing important statistics about the 
training corpora <cit.>.
As can be expected, the more data a model is trained on, the better the representations
are at transferring to other NLP problems.
Training such models on massive data sources, like Common Crawl, can be cumbersome and
many NLP practitioners prefer to use publicly available pre-trained word
vectors over training the models by themselves.
In this work, we provide new pre-trained word vectors that show consistent improvement
over the currently available ones, making them potentially very useful to a wide community
of researchers.















We show that several modifications of the standard word2vec training pipeline
significantly improves the quality of the resulting word vectors.
We focus mainly on known modifications and data 
pre-processing strategies that are rarely used together:
the position dependent features introduced by mnih2013learning,
the phrase representations used in mikolov2013distributed and the use of subword information <cit.>.

We measure their quality on standard benchmarks: syntactic, semantic and phrase-based
analogies <cit.>,
rare words dataset <cit.>, and as features in a question answering pipeline
on Squad question answering dataset <cit.>.



§ MODEL DESCRIPTION


In this section, we briefly describe the cbow model as it was used in word2vec, 
and then explain several known improvements to learn richer word representations.



 §.§ Standard cbow model



The cbow model as used in mikolov2013efficient learns word representations
by predicting a word according to its context. 
The context is defined as a symmetric window containing all the surrounding words.
More precisely, given a sequence of T words w_1,…,w_T, the objective of 
the cbow model is to maximize the log-likelihood of the probability of 
the words given their surrounding, i.e.:

    ∑_t=1^T log p(w_t | C_t),

where C_t is the context of the t-th word, e.g., the 
words w_t-c,… w_t-1, w_t+1,…, w_t+c for a context window of size 2c.
For now on, we assume that we have access to a scoring function between a word w and its context C, denoted by s(w, C). This scoring function
will be later parametrized by the word vectors, or representations.
A natural candidate for the conditional 
probability in Eq. <ref> is a softmax function over the scores of a context
and words in the vocabulary. 
This choice is however impractical for large vocabulary.
An alternative is
to replace this probability by independent binary classifiers over words. 
The correct word is learned in contrast with a set of sampled negative candidates.
More precisely, the conditional probability of a word w 
given its context C in Eq. (<ref>) is replaced by the following quantity:

    log(1+ e^-s(w,C)) + ∑_n∈ N_Clog(1+e^s(n,C)),

where N_C is a set of negative examples sampled from the vocabulary. 
The objective function maximized by the cbow model is obtained by replacing the 
log probability in Eq. (<ref>) by the quantity defined in Eq. (<ref>), i.e.:

    ∑_t=1^T [ log(1+ e^-s(w_t,C_t)) + ∑_n∈ N_C_tlog(1+e^s(n,C_t))].

A natural parametrization for this model is to represent each word w by a vector v_w. 
Similarly, 
a context is represented by the average of word vectors u_w' of each word w' in its window.
The scoring function is simply the dot product between these two quantities, i.e.,

    s(w,C) = 1/|C|∑_w'∈ C u_w'^Tv_w.

Note that different parametrizations are used for the words in a context and the predicted word.



  
Word subsampling.
The word frequency distribution in a standard text corpus follows a Zipf distribution, which implies that most of the words belongs to small subset of the entire vocabulary <cit.>.
Considering all the occurences of words equally would lead to overfit the parameters of the model on the representation of the most frequent words, while underfitting on the rest.
A common strategy introduced in mikolov2013efficient is to subsample frequent words, 
with the following probability p_disc of discarding a word:

    p_disc(w) = 1 - √(t / f_w)

where f_w is  the frequency of the word w, and t>0 is a parameter.











 §.§ Position-dependent Weighting



The context vector described above is simply the average of the word vectors contained in it. 
This representation is oblivious to the position of each word. 
Explicitly encoding a representation for both a word and its position would be impractical and prone to overfitting.
A simple yet effective solution introduced in the context of word representation by mnih2013learning is to learn position representations
and use them to reweight the word vectors.
This position dependent weighting offers a richer context representation at a minimal 
computational cost.

Each position p in a context window is associated with a vector d_p. The context vector
is then the average of the context words reweighted by their position vectors.
More precisely, denoting by P the set of 
relative positions [-c,…,-1,1,…,c] in the
context window, the context vector v_C of the word w_t is:

    v_C = ∑_p∈ P d_p ⊙ u_t+p,

where ⊙ is the pointwise multiplication of vectors.



 §.§ Phrase representations


The original cbow model is only based on unigrams, which is 
insensitive to the word order.
We enrich this model with word n-grams to capture richer information.
Directly incorporating the n-grams in the models is quite challenging as
it clutters the models with uninformative content due to huge increase
of the number of the parameters.
Instead, we follow the approach of mikolov2013distributed
where n-grams are selected by iteratively applying a mutual information
criterion to bigrams.
Then, in a data pre-processing step we merge the words in a selected n-gram into a single token.

For example, words with high mutual information like "New York"
are merged in a bigram token, "New_York".
This  pre-processing step is repeated several times to form longer n-gram tokens,
like "New_York_City" or "New_York_University". 
In practice, we repeat this process 5 - 6 times to build tokens representing longer ngrams.
We used the word2phrase tool from the word2vec project[<https://github.com/tmikolov/word2vec>].
Note that unigrams with high mutual information are merged
only with a probability of 50%, thus we still keep significant number of unigram occurrences.
Interstingly, even if the phrase representations are not further used in an application,
they effectively improve the quality of the word vectors, as is shown in the experimental section.





















 §.§ Subword information


Standard word vectors ignore word internal structure that contains
rich information. This information could be useful for computing representations
of rare or mispelled words, as well as for mophologically rich languages like Finnish or Turkish.
A simple yet effective approach is to enrich the word vectors with a bag of
character n-gram vectors that is either derived from the singular value
decomposition of the co-occurence matrix <cit.> or directly
learned from a large corpus of data <cit.>.
In the latter, each word is decomposed into its character n-grams N and each n-gram n
is represented by a vector x_n. The word vector is then simply the sum of both
representations, i.e.:

    v_w + 1/|N|∑_n∈ N x_n.

In practice, the set of n-grams N is restricted to the n-grams with 3 to 6 characters.
Storing all of these additional vectors is memory demanding.
We use the hashing trick to circumvent this issue <cit.>.



§ TRAINING DATA


We used several sources of text data that are publicly available and the Gigaword
dataset, as described in Table 1. In particular, we used English Wikipedia from June 2017,
from which we used the meta pages archive which resulted in a text corpus with more than 9 billion words [<https://dumps.wikimedia.org/enwiki/latest/>].
Further, we used all news datasets from <statmt.org> from years 2007 - 2016, the UMBC corpus <cit.>, the English Gigaword,
and Common Crawl from May 2017[<https://commoncrawl.org/2017/06>].

In case of the Common Crawl, we wrote a simple data extractor based on a unigram language model that
retrieves the documents written in English and discards low quality data. The same approach
can be in fact used to extract text data for many other languages from Common Crawl.




We decided to perform no complex data normalization or pre-processing, as we want the resulting
word vectors to be very easily used by a wide community (the text normalization can be done on top of the published word vectors
as a post-processing step). We only used a publicly available tokenizer.perl
script from the Moses MT project[<https://github.com/moses-smt>]. 
We observed that de-duplicating large text training corpora, especially Common Crawl,
significantly improves the quality of the resulting word vectors.





























§ RESULTS


Further we report results for models trained on either the Common Crawl, or on a combination of the Wikipedia, Statmt News, UMBC and Gigaword.
This is comparable to corpora that other models that attempted to improve upon word2vec were trained on, notably the GloVe model from
the Stanford NLP group <cit.>. Although a careful analysis performed in levy2015improving shows that the original
word2vec is faster to train, produces more accurate models and takes significantly less memory than the GloVe algorithm,
the availability of large pre-trained GloVe models proved to be a useful resource for many researchers who do not have time
to train their own model on very large dataset like the Common Crawl.

We used the cbow architecture described in Section <ref> with window size 5 for the baseline models
and window size 15 for the models that learn position-dependent weights (described in Section <ref>).
We used 10 negative examples for training with the negative sampling
and threshold for subsampling frequent words set to t = 10^-5.

In Table <ref> we report results on the word analogies from mikolov2013efficient using baseline cbow model trained on Common
Crawl with de-duplicated sentences, with phrases (we used 6 iterations of building the phrases by merging bigrams with high mutual information), and
with the position-dependent weighting as used in mnih2013learning. The training itself
took three days on a single multi-core machine.



In Table <ref>, we can see comparison between cbow as implemented in the fastText library <cit.> and the GloVe
models trained on comparable corpora. The 87% accuracy on the word analogy tasks is to our knowledge the best published result so far by a large margin,
and much better than existing GloVe models which were trained on comparable corpora. We improved this result further to 88.5% accuracy by adding the sub-word features.

We also report very strong performance on the Rare Words dataset <cit.>,
again outperforming GloVe models by a large margin. Finally, we replaced the GloVe pre-trained vectors with the new fastText vectors in a
question answering system trained on the Squad dataset <cit.>. In a setup that is further described in chen2017reading, we did observe
significant improvement of the accuracy.



The models trained on the Wikipedia and News corpora, and on the Common Crawl, were published at the <fasttext.cc> website
and are available to the NLP researchers. Further, we did experiment with the phrase-based analogy dataset
introduced in mikolov2013distributed, and achieved 88% accuracy using the model trained on Crawl, which again
is to our knowledge the new state of the art result. We plan to release the model containing all the phrases in the near future.



Finally in Table <ref>, we use a script provided by conneau2017supervised to
measure the influence of different pre-trained word vector models on several text classification tasks (MRPC, MR CR, SUBJ, MPQA, SST and TREC).
We performed the classification using the standard fastText toolkit running in a supervised mode <cit.>, using the pre-trained models
to initialize the classifier. Overall, the new fastText word vectors result in superior text classification performance.



§ DISCUSSION


In this work, we have focused on providing very high quality set of pre-trained word and phrase vector representations.
Our findings indicate that improvements can be achieved by training well-known algorithms on very large text datasets, and that
using certain tricks can provide further gains in quality. Notably, we have found it very important to de-duplicate sentences
in large corpora such as the Common Crawl before training the models. Next, we have used an algorithm for building the phrases
in a pre-processing step. Finally, adding the position-dependent weights and subword features to the cbow model architecture gave us the final boost
of accuracy. The models described in this paper are freely available to researchers and engineers at the fastText webpage, and we hope
that these will be useful in various projects that use textual data.









§ ACKNOWLEDGEMENTS


We thank Marco Baroni and German Kruszewski for useful discussions and suggestions, Adam Fisch for help with the experiments on Squad
dataset, and Qun Luo for suggesting to use the position-dependent weighting at the word2vec discussion forum.



§ BIBLIOGRAPHICAL REFERENCES


lrec







