1

A WL-SPPIM Semantic Model for Document
Classification
First A. Ming Li, Second B. Peilun Xiao, and Third C. Ju Zhang
Abstract—In this paper, we explore SPPIM-based text classification method, and the experiment reveals that the SPPIM
method is equal to or even superior than SGNS method in text classification task on three international and standard text
datasets, namely 20newsgroups, Reuters52 and WebKB. Comparing to SGNS, although SPPMI provides a better solution, it is
not necessarily better than SGNS in text classification tasks.. Based on our analysis, SGNS takes into the consideration of
weight calculation during decomposition process, so it has better performance than SPPIM in some standard datasets. Inspired
by this, we propose a WL-SPPIM semantic model based on SPPIM model, and experiment shows that WL-SPPIM approach
has better classification and higher scalability in the text classification task compared with LDA, SGNS and SPPIM approaches.
Index Terms—LDA; SPPIM; word embedding; low frequency;document classification

——————————  ——————————

1 INTRODUCTION

D

istribution of semantic vectors is widely used in text
semantic expression, including text classification, text
clustering, semantic retrieval, automatic question and
answer, dictionary generation, semantic disambiguation,
query expansion, text advertisements and machine
translation, especially for measuring semantic relevance[1,
2]. We divided the DSMs into two categories, one we
called count-based models, many traditional DSMs
belong to this category, the other category we call
prediction-based models, which are based on neural
embedding
Among the traditional count-based models the best
know is Latent Semantic Analysis. LSA is a low
dimensional semantic space for texts, and LSA derive the
document vector by the use of co-occurrence information
between words[3]. More recently, LDA has received more
and more extensive attention, as a semantic model of
DSMs[4, 5]. LDA is a three-layer Bayesian probability
model proposed by Blei et al in 2003, which contains the
three layers structure of document, topic and word.
Document to topic subjects to Dirichlet distribution, topic
to word subjects to polynomial distribution. LDA
semantic model usually shows very good performance on
NLP tasks, partly because it projects the document into a
low-dimensional topic semantic space. Word frequencies
determine the topic of calculated and deducted by LDA
largely, which leads less frequent but important topics
can not be effectively calculated.
————————————————

 F. A. Author is with High performance computing application R&D
Center, Chongqing Institute of Green and Intelligent Technology, Chinese
Academy of Sciences, Chongqing, China & University of Chinese Academy
of Sciences, Beijing, China & Center for Speech and Language
Technology,Research Institute of Information Technology, Tsinghua
University,Beijing, China; E-mail: liming@cigit.ac.cn.
 S.B. Author is an undergraduate in Beijing University of Posts and
Telecommunications. This work was done when he was a visiting student at
Tsinghua University; Email:xiaopli@cslt.riit.tsinghua.edu.cn.
 T. C. Author is with with High performance computing application R&D
Center, Chongqing Institute of Green and Intelligent Technology, Chinese
Academy of Sciences Chongqing, China; E-mail: Zhangju@cigit.ac.cn.

Pointwise mutual information(PMI) has been
extensively uesd as count-based model in distributional
sematic models. Pointwise Mutual Information (PMI) is
the popular word co-occurrence based measure[6, 7]. PMI
has a well-known tendency that it calculates too high
scores for low frequency words[2]. To solve the limitation
fo PMI, many variants of it have been proposed, PPMI is
the simple one of the variants, in which all PMI values
that below zero will be set to zero[8]. Bullinaria and
Levydem demonstrate that PPMI outperforms various
other weighting methods ， when measuring semantic
similarity through word-context matrices.[9].
Traditional distributional Semantic Models achieve
considerable effective effects on various NLP tasks,
including semantic relevance and text classification.
The last few years have seen the development of
prediction-based neural embedding models in which
words are embedded into a low dimensional space. Word
embedding models can efficiently learn word vector from
a large number of unstructured text, and can effectively
reflect the syntactic or semantic relations between
words[10]. The initial pioneering research work was
started by Bengio and his colleagues, who generated
word vectors in the study of the neural language
model[11] and a number of subsequent research work
including various word embedding models and efficient
learning algorithms[12-16]. In particular, We notice a
conclusion that the SGNS(skip-gram with negativesampling)model which is efficient to provides
competitive results on various NLP tasks, which is
concluded in a sequence of papers by Mikolov and
colleagues[14, 17]. The SGNS model maximizes the
conditional probability of the observed contexts given the
current word when scanning through the corpus,
however it is not clear what information the embedding
vectors really convey, so prediction-based neural
embeddings are considered opaque.
The researchers have different conclusions on the
various types of distributed semantic models applying to
the performance of NLP tasks, some of these conclusions

2

are the same, some are the opposite, some conclusions
depend on the distributed semantic models application of
specific NLP tasks . Some of the researchers have found
that predictive neural embedding models have better
performance than traditional counting models in
calculating word similarity[9, 18], some researchers have
published papers that show the traditional counting
models have better performancethan than predictionbased neural embedding models on some datasets of
specific NLP tasks[19-20].
So far no study has been conducted in this field about
comparing the performance of count-based models and
prediction neural embedding models, such as LDA,
PIM(or other PMI variants) and SGNS which are
representative distributional semantic models on the
document classification. In this paper, We make a deep
research and compare the performance of LDA, SGNS,
SPPIM distributional semantic models in text
categorization. The Experiment shows that whether the
TSVD method or the ISVD method get the poor
performance than the PPMI method in the similaraty task
in the low frequency range[21]. Especially, the ISVD
method get the worst results. The paper analyzed these
results as that ISVD removed the latent dimension vectors
which have the largest variance, that are no doubt the
most important dimensions for fairly low frequent item.
Inspired by the paper insight, we propose a WL-SPPIM
semantic model based on SPPIM model empirically.
Different with SPPIM: 1) We make some changes on one
term in the formula that produces the PMI matric, by
replacing the P(c) with p 3 4 (c ) ; 2) we reweight the
probility of the low frequency word by merging the
probability of the high frequency words into the probility
of the low frequency words. The experimental results
show that, on the document classification, WL-SPPIM
yields better results than LDA, SGNS, SPPIM on the
accuracy and expansibility.
The rest of the paper is structured as follows: Section 2
introduces the LDA, SGNS, SPPIM semantic models for
document classification; Section 3 describes the proposed
WL-SPPIM semantic model for document classification;
Section 4 presents the experiments; Section 5 discusses
the result; and Section 6 conclusions the paper and
suggests the future work direction.

2 BACKGROUND
2.1 LDA-based Semantic Model
LDA is not only a topic model, but also a “text-topicword" three-level Bayesian model. The text reflects mixes
distribution of topic, while the topic demonstrates the
probability distribution of words. A document contains
many topics, and each word in the document is generated
by one topic. As for each word in the document

d   w1 , w2 ,K wN  , LDA model chooses a topic  z ,
topic set  based on the multinomial distribution
i

Multi   which is determined by related variable  , in
z
which i represents the number of selected topic, and  is

consistent with the Dirichlet Distribution Dir   . After
the topic  z is selected, the word wi is generated by
i

 

multionmial distribution Multi  z
i
by

z

i

which is defined

, the generation of text is summarized by formula

(1) - (3).

 ~ Dir ( )
zi ~ Multi ( )
wi ~ Multi (  z i )

(1)
(2)
(3)

Text feature vectors are generated by adopting LDA to
extract text features of documents in low dimensional
space. These feature vectors can be used for building
classification models and classification. This paper
extracts text features of LDA according to following steps:
1) set the size of vocabulary V and topic set  ; 2) input
all documents of training set (regardless of category) into
LDA model, training model parameters  and  ; 3)
LDA models obtained from training are used in reasoning
to calculate P  | d . In general, the maximum point 
of P  | d is used as feature vector of the document,









 MAP  , the maximum posteriori probability
 : f  d    MAP  d   arg max P ( | d ) .

namely
point



2.2 SPPIM semantic model
The traditional way to represent words in the
distributional approach is to construct a high dimensional
sparse matrix M , where each row represents a word w
and each column represents a context c . The value of
each matrix cell
the word

wi

M ij

represents the association between

and the context

cj .

A popular measure of

thisassociation is pointwise mutual information(PMI)[6].
PMI is calculated empirically like:
#(w, c) D
(4)
P M I ( w , c )  lo g
# ( w )g# (c )
D is the number of observed word and context pairs.
In terms of negative PMI values may not positively
contribute to model performance and sparser matrices are
more computationally tractable[2], a sparse and
consistent alternative is to use PPMI matric. In PPMI
matric, all negative values which are deduced from PMI
matric are replaced by 0:
(5)
PPMI ( w, c )  max( PMI ( w, c ), 0)
SPPMI can be generalized to an additional coutoff
parameter k , analysising by Levy and Goldberg shows
hat SGNS is equivalent to a shifted version of the PPMI
method, where all values get shifted by a factor log k [22]:

SPPMI ( w, c)  max( PMI ( w, c)  log k , 0) (6)
In the SPPMI model, word distributions are projected
into high dimensional sparse space. Distributional
semantic models can get better performance, when
applying a variety of transformations to the original
vector, such as by reweighting to the text and applying
the dimensionality reduction technique. So we apply SVD
to the SPPMI matrix in order to obtain low-dimensional

3

embeddings.When the SPPIM-SVD matrix produces the
word vector, the document vector vi can be computed
also by the simple average pooling approach.

2.3 SGNS Semantic Mode
Prediction-based neural embedding models such as the
skip-gram model and continuous bag-of-words model
have become the standard for word modeling[14, 17]. In
the skip-gram model, for a word w  VW and a context

c  VC , their

w R

d

embedding vectors are represented as
d

and c  R respectively, where d is the
embedding’s dimensionality. The embedding vectors of
the words in VW and the context words in VC compose
the word and context embedding matrices

W

and

C . It

w  VW and each context
r
r
vectors w and c , the words

seeks to represent each

c  VC

as d-dimensional

will have similar vector representations, if the words are
similar.
Levy and Goldberg made an interesting connection
between two models: a traditional count-based model on
PMI(pointwise mutual information) and a predictionbased neural embedding model, namely the SGNS(skipgram model with negative sampling)[22]. The interesting
result is that the SGNS model is equivalent to a shifted
version of the PMI method, where all values get shifted
by a factor of log k . Levy and Goldberg described the
matrix M that SGNS is factorizing:
r r
(7)
M ijSGNS  Wi  C j  wi  c j  PMI ( wi , c j )  log k
The meaning of the document can be considered a
summary of the meaning of all the words in the
document. We use the Word2vec tool to generate the
word vector first, and then generate the document vector
by averaging pooling the word vector. We use

ci , j to

denote the word vector of the j th word in the document

i , the
vi 

1
Ji

document vector
Ji

c
j 1

i, j

vi can

be

calculated

as:

, Where J i is the number of words in the

The SPPIM underperforms than the SGNS on the one
dataset on text classification task(see section 4), one of the
main reason is that SGNS performs weighted matrix
factorization. Inspired by this, we propose the WL-SPPIM
approach. The probabilities of the low frequency words
are generally underestimated in terms of the lack of
occurrences in the training datan, and estimating the
probabilities of words that absent in corpus is simply
impossible. However, these words are often important
entity names that should be emphasized, so we have a
idea that transform some information from high
frequency words to enhance the weight of the lowfrequency
words.
Given
a
set
of
words

W   x1 , x2 ,K, xm  to be enhanced, for

each word

xi  W , a set of words Si   y1 , y2 , K yn  that are similar

to xi is manually selected from the training data. The
similarity can be defined with semantic. We assume that,
for each y j  Si , if there exist an n-gram of
training corpora, the corresponding

y j in

n-gram

of

the

xi

should also have a relative higher
probability
of
appearance. In general, TF -IDF method is adopted
for
the weight
design of features[23].
N
is
the
T F I D F t , d  T F t , d  lo g









DF t 

2

calculation formula of word frequency-inverse document
frequency; TF  t , d  refers to the frequency of word t in
document d ,

N is total document number, and DF  t  is
the number of other documents containing feature word t .
But TF -IDF lacks the ability to distinguish feature items
in different categories. We designed new weight
calculation method based on TF -IDF technology.
represents the
Nc
W

t | c 

 T F  t | c   lo g 2

ability of feature item

c , c  c1 , c2 , ck

of feature item

t



DF t |c

t



to distinguish category

and TF  t | c  represents the frequency
in category

c,

c , Nc

is the number of

 

document.

documents in category

3 WL-SPPIM SEMANTIC MODEL

the number of documents containing feature item t
which are included in the class that not belongs to c . The
value of W  t | c  is larger, the ability of feature item to

The Shifted PPMI performs slightly better than SGNS
derived vectors on two datasets , but performs poorly
than the latter on one dataset of text classification
tasks(see section 4).
According to the algorithm illustrated in[17], the
negative contexts are sampled according to the formula

# c 3 4 instead of the unigram distribution
z
# ( c ) , so we modify the SPPIM formulation :

p 3 4 (c ) 
z

SPPMI ( w, c)  max( PMI ( w, c)  log k 3 4 , 0)

(8)

distinguish category

c

and DF t | c

represents the

is the stronger. W  t | c  measures

the ability of feature word to distinguish categories from
global perspective.
Then we adjust the weight of the low-frequency
according to the following equation:


W ( xi | c )
(9)


w x i  w y j  ln
2 

W ( xi | ci ) 
  ci  c

Where xi presents the low-frequency word.

4

W  xi | c  indicates the ability of

category

xi to

TABLE 1
DATASETS STATISTICS

distinguish

ci , TF  xi | ci  represents the frequency of xi in

category ci , Nc is the number of documents in category ci ,
i
DF  xi | ci  is the number of documents containing word

Datasets

Train

Test

Word

Stop

20ng

11293

7528

93864

no

R52

6532

2568

26284

no

WebKB

2756

1375

7770

no

xi which are included in the class that not belongs to c .

4.2 Experiments
The first experiment investigates the performances of
document classification methods LDA, SGNS, SPPIM and
i
WL-SPPIM when the dimensionality of document vector
value is too small to enhance the recognition performance, is different. In LDA method, dimensionality denotes the
number of topic in topic set  ; in SGNS, SPPIM and WLthis threshold will be assigned to it .
SPPIM, dimensionality refers to the dimension of space.
The document classification performance of document
4 EXPERIMENT
vectors generated by the above four semantic models in
20News, Reuters52 and WebKB datasets are tested. Figure
4.1 Database and configurations
We experiment all models on English corpus, 1-3 shows classification performance of text features
preprocessing by removing single characters and generated by above 4 models in 20News, R52 and WebKB,
alphabetic characters, and converting letters to lowercase, respectively, the abscissa refers to the dimension of
stemming word roots, sentence splitting, and document vector and ordinate the accuracy of document
tokenization. The experiments were conducted with three classification on the three datasets.
SGNS has a natural hyper parameter k , which is the
datasets:20 Newsgroups dataset that was originally
1
number
of negative samples, which affects the value that
collected by Ken Lang , WebKB dataset that was collected
2
SGNS
is
trying
to
optimize
for
each
by the
project of the CMU , and Reuters dataset
3
he
shift
caused
by
.
T
can
(
w
,
c
)
:
PMI
(
w
,
c
)

log
k
k

1
published by DavidD Lewis . Table 3 lists the statistics of
be applied to distributional methods through shifted
these three datasets.
20Newsgroups corpus contain 20 categories of English PPMI. SPPMI achieve good performance with lower
news text, which contains a total of 18846 Document. In values of k , what is due to the fact that only positive
addition,in order to improve the reliability of experiment, values are retained, and high values of k may cause too
all repeat documents and some news heads are removed, much loss of information[22]. Empirically, we take
which left 11293 and 7528 documents to the training data experiment with values of k  5 .
At first, We experiment these models with different
and testing data. Original WebKB corpus contains about
classifiers
on 20newsgroups, and the results about
8,300 English websites, which can be divided into 7
categories. Selecting the most commonly used 4 major classification accuracy are depicted in Table 2. We
categories, including student, faculty, course and project conduct this experiment with 50-dimension on the word
categories, this text subset is called WebKB-4. In addition, vector in the SGNS and the topics in the LDA, then the
in order to improve the reliability of experiment, some dimensions of the document vectors will be 50-dimension
repeat documents are removed, which left 2756 and 1375 with four approaches. We observe that KNN method
documents to the training set and testing set. The Reuters outperforms the other two, when the three classifiers are
dataset is a collection of Reuters Newswire in 1987. We compared, which the result is showed in Table 2.
have 8 of the 10 most frequent classes and 52 of the Nevertheless, KNN is a nonparametric method, which is
original 90, namely R8 and R52. R52 is a totally 4M corpus not suitable for large corpus, so we choose SVM as the
which contains 3M text (6532 documents) for training, classifier which will be used in the following experiments.
and 1M text(2568 documents) for testing.
TABLE 2
We use the word2vec tool provided by Googleused to
CLASSIFICATION WITH DIFFERENT CLASSIFIERS ON 20NEWSGROUPS
4
train the skip-gram word vector . LDA training and
learning tools invented by Blei is used to generate
Classifier
LDA
SNGS(W2V)
SPPIM
WL-SPPIM
document vector5.
We conducted text classification
NB
61.47%
63.69%
62.71%
66.33%
experiments on three classifiers, including Bayesian, KNN,
and SVM which all are build by the scikit-learn tool6.
KNN
78.21%
75.15%
72.10%
77.63%
In addition, a threshold
is set to guarantee that
W ( xi | c)
is not less than this threshold, if the
2
W
(
x
|
c
)
 c c i i

SVM
1
http://www.cs.cmu.edu/afs/cs.cmu.edu/project/theo20/www/data/news20.html
2
http://web.ist.ult.pt/~acardoso
3
https://kdd.ics.uci.edu/databases/reuters21578/reuters21578.html
4
https://code.google.com/p/word2vec
5
http://www.cs.princeton.edu/~blei/lda-c
6
http://scikit-learn.org/dev/modules/svm.html#svm

71.67%

75.15%

72.94%

77.98%

5

dataset to test the performance of the classification task
complexity of the four models in each classification task.
For each classification task, LDA, SPPIM and WL-SPPIM
should retrain parameters in corresponding training
corpus, while word vector generated by word2vector
tools does not depend on specific text field, so SGNS
model does not retrain parameters. In the Figure 1-3, the
abscissa represents different complexity on document
classification, and ordinate the accuracy of classification.

5 EXPERIMENTAL RESULTS AND ANALYSIS

Fig. 1.Performance on 20-Newsgroups

Fig. 2. Performance on Reuters52

Fig. 3. Performance on WebKB

The second experiment compared the performances of
the four models on document classification of different
complexity. Figure 4 shows that text feature dimension is
set at 200 dimensions, and classification complexity is
increased from two-level classification to 20-level
classification in 20Newsgroups dataset, and from twolevel classification to 4-level classification in WebKB

We evaluate LDA, SGNS, SPPIM, WL-SPPIM on the text
classification task. The comparison of these four models
in terms of the text classification accuracy is seen in Fig 1Fig3 , the dimensionality of the document vectors varies
from 10 to 300 in our experiment . Fig1-Fig3 shows the
WL-SPPIM model achieves its best performs on all the
datasets, WL-SPPIM outperforms SPPIM consistently.
And the SPPIM is subtly better than SGNS on R52 and
WebKB datasets, the accuracy of SPPIM and SGNS are
very close on 20newsgroups datasets. WL-SPPIM
significantly outperforms SPPIM and SGNS, verifying
that re-weighting the weight of the word vector could
improve the accuracy of the document classification.
Given the similar performance of SPPIM and SGNS in the
text classification task, the superiority of WL-SPPIM is
obvious. The accuracy of LDA is much lower than that of
other methods, which may be because LDA performs
better with long documents[24]. It can be learned that the
accuracy of the four models in classification improves
with the increase of document vector dimensions,
indicating document vector dimension is larger, the
classification accuracy is better. However, the
classification performance of LDA in WebKB data
decreases sharply with the increase of feature
dimensions, which is attributed to the lack of long text
training corpus in the field, difficult to meet the
optimization task of LDA n high dimensional vector
model.
As to the effects of different category, one can observe
that the performance of WL-SPPIM remains stable with
different categories from Figure 4 The accuracy of LDA
is much lower than that of other methods, which may be
due to uneven distribution of topic vector in various
categories and thereby leading to differences in various
categories as LDA is greatly influenced by
model initial value in the establishment of topic model.
As the generation of document vector by SGNS
combination involves the process of mean smoothing, so
it is less affected by categories changing and the
classification
performance
is
stable.
Similarly,
reweighting the low frequency words in WL-SPPIM
model is equivalent to smoothing techniques, so multiclassification experiments shows better classification
performance and higher stability.

6

6 CONCLUSION

Fig. 4. Experimental results with various numbers of
category on 20-newsgroups

In this paper, we verify the discovery in[20, 25] on the
text classification task, what the traditional count-based
models achieve better performance than predictionbased neural embedding models on some specific NLP
tasks. Furthermore, we propose a method based on WLSPPIM semantic model for the document classification,
and investigate the characteristics of LDA, SGNS, SPPIM
and WL-SPPIM sematic models on the semantic source
and semantic generation method in detail, and analyze
the advantages of WL-SPPIM semantic model on the
document classification. Experiments shows that the WLSPPIM model is significantly superior than other three
methods in classification accuracy, complexity, scalability
on text classification tasks. In future work we will
investigate distributions semantic clustering for WLSPPIM model.

ACKNOWLEDGMENT
This research was supported by the National Science
Foundation of China (NSFC) under the project
No.61672488, Ministry of science and Technology “ Key
technologies of educational cloud ”
project
No.2013BAH72B01, Chongqing Science and Technology
Commission project No.cstc2013jjys4000. It was also
partially supported by the Magicalthink Ltd.

REFERENCES
[1]

Fig. 5. Experimental results with various numbers of
category on WebKB

From the results revealed in Fig4 and Fig5, we can
observe that
the proposed method produce good
accuracy on the 20newsgroups dataset and R52 dataset,
but not satisfied on the WebKB dataset. Improving the
weight of low-frequency words does not enhance the
classification performance of WL-SPPIM model in the
WebKB dataset, which may be because the WebKB
dataset is relatively small, so low frequency and high
frequency words meeting the requirements of similar-pair
are very few.This result confirms the effectiveness and
high performance of proposed method. This method
could be applied on semantic tasks in specific domain
where low-frequency words need to be enhanced.
The experimental results reveal that our count-based
model achieve better performance
than predictionbased neural
embedding model on the document
classification task. Furthermore, these experiments not
only verify the classification performance of WL-SPPIM,
but also the generalization ability of WL-SPPIM.

S. Dumais, “Data-driven approaches to information access,”
Cognitive Science, vol. 27, no. 3, pp. 491-524, 2003.
[2] P. D. Turney, and P. Pantel, “From frequency to meaning:
Vector space models of semantics,” Journal of artificial
intelligence research, vol. 37, pp. 141-188, 2010.
[3] T. K. Landauer, and S. T. Dumais, “A solution to Plato's problem:
The latent semantic analysis theory of acquisition, induction,
and representation of knowledge,” Psychological review, vol.
104, no. 2, pp. 211, 1997.
[4] D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent dirichlet
allocation,” Journal of machine Learning research, vol. 3, no.
Jan, pp. 993-1022, 2003.
[5] T. L. Griffiths, M. Steyvers, and J. B. Tenenbaum, “Topics in
semantic representation,” Psychological review, vol. 114, no. 2,
pp. 211, 2007.
[6] K. W. Church, and P. Hanks, “Word association norms, mutual
information, and lexicography,” Computational linguistics, vol.
16, no. 1, pp. 22-29, 1990.
[7] P. D. Turney, "Mining the web for synonyms: PMI-IR versus
LSA on TOEFL." pp. 491-502.
[8] Y. Niwa, and Y. Nitta, "Co-occurrence vectors from corpora vs.
distance vectors from dictionaries." pp.304-309.
[9] T. Mikolov, W.-t. Yih, and G. Zweig, "Linguistic Regularities in
Continuous Space Word Representations." pp. 746-751.
[10] G. E. Hinton, J. L. Mcclelland, and D. E. Rumelhart,
"Distributed representations, Parallel distributed processing:
explorations in the microstructure of cognition, vol. 1:
foundations," MIT Press, Cambridge, MA, 1986.
[11] Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin, “A neural
probabilistic language model,” Journal of machine learning
research, vol. 3, no. Feb, pp. 1137-1155, 2003.

7

[12] A. Mnih, and G. E. Hinton, "A scalable hierarchical distributed
language model." pp. 1081-1088.
[13] A. Mnih, and G. Hinton, "Three new graphical models for
statistical language modelling." pp. 641-648.
[14] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient
estimation of word representations in vector space,” arXiv
preprint arXiv:1301.3781, 2013.
[15] T. Mikolov, “Statistical language models based on neural
networks,” Presentation at Google, Mountain View, 2nd April,
2012.
[16] R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu,
and P. Kuksa, “Natural language processing (almost) from
scratch,” Journal of Machine Learning Research, vol. 12, no.
Aug, pp. 2493-2537, 2011.
[17] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean,
"Distributed representations of words and phrases and their
compositionality." pp. 3111-3119.
[18] M. Baroni, G. Dinu, and G. Kruszewski, "Don't count, predict! A
systematic comparison of context-counting vs. contextpredicting semantic vectors." pp. 238-247.
[19] F. Asr, J. Willits, and M. Jones, "Comparing predictive and cooccurrence based models of lexical semantics trained on childdirected speech."
[20] O. Levy, Y. Goldberg, and I. Ramat-Gan, "Linguistic
Regularities in Sparse and Explicit Word Representations." pp.
171-180.
[21] M. Sahlgren, and A. Lenci, “The Effects of Data Size and
Frequency Range on Distributional Semantic Models,” arXiv
preprint arXiv:1609.08293, 2016.
[22] O. Levy, and Y. Goldberg, "Neural word embedding as implicit
matrix factorization." pp. 2177-2185.
[23] W. Zhang, T. Yoshida, and X. Tang, “A comparative study of TF*
IDF, LSI and multi-words for text classification,” Expert
Systems with Applications, vol. 38, no. 3, pp. 2758-2765, 2011.
[24] J. Tang, Z. Meng, X. Nguyen, Q. Mei, and M. Zhang,
"Understanding the Limiting Factors of Topic Modeling via
Posterior Contraction Analysis." pp. 190-198.
[25] A. Mnih, and K. Kavukcuoglu, "Learning word embeddings
efficiently with noise-contrastive estimation." pp. 2265-2273.

