Why word2vec performs much worst than both CountVectorizer and TfidfVectorizer? [Text classification] I'm following this guide to try creating both binary classifier and multi-label classifier using MeanEmbeddingVectorizer and TfidfEmbeddingVectorizer shown in the guide above as inputs.

Both embedding vectorizers are created by first, initiating w2v from documents using gensim library, then do vector mapping to all given words in a document and vectorizes them by taking the mean of all the vectors corresponding to individual words.

After this step has completed, I tried using these embedding vectorizers as inputs into several model such as OneVsRestClassifier(SVC), RandomForestClassifier and ExtraTreesClassifier. Hence, all of the models performs worst than my expectation, with respect to what is shown in the guide.

These are the accuracies from each models (binary classifer) :

randomF_countVect: 0.8898
extraT_countVect: 0.8855
extraT_tfidf: 0.8766
randomF_tfidf: 0.8701
svc_tfidf: 0.8646
svc_countVect: 0.8604
ExtraTrees_w2v: 0.7285
ExtraTrees_w2v_tfidf: 0.7241


Multi-label classifier also produced similar result.

I'm not sure that I've done wrong. 

Note that I'm working with very small documents. In which each document consists of short text (a sentence or two) and they're non-English documents. In total, whole documents only has 1163 unique words. Below is my code. Could someone please light me up?

Word2vec initiation

model = Word2Vec([my_tokenizer(item) for item in df['text']], size=100, window=5, min_count=1, workers=2)
w2v = {w: vec for w, vec in zip(model.wv.index2word, model.wv.syn0)}


Models

svc = Pipeline([("count_vectorizer", vectorizer), ("OneVSRest svc linear", OneVsRestClassifier(SVC(kernel='linear')))])
svc_tfidf = Pipeline([("tfidf_vectorizer", tf_vectorizer), ("OneVSRest svc linear", OneVsRestClassifier(SVC(kernel='linear')))])
randomF = Pipeline([("count_vectorizer", vectorizer), ("RandomForestClassifier", RandomForestClassifier(n_estimators=100))])
randomF_tfidf = Pipeline([("tfidf_vectorizer", tf_vectorizer), ("RandomForestClassifier", RandomForestClassifier(n_estimators=100))])
extraT = Pipeline([("count_vectorizer", vectorizer), ("ExtraTreesClassifier", ExtraTreesClassifier(n_estimators=100))])
extraT_tfidf = Pipeline([("tfidf_vectorizer", tf_vectorizer), ("ExtraTreesClassifier", ExtraTreesClassifier(n_estimators=100))])
etree_w2v = Pipeline([("word2vec vectorizer", MeanEmbeddingVectorizer(w2v)), ("word2vec extra trees", ExtraTreesClassifier(n_estimators=100))])
etree_w2v_tfidf = Pipeline([("tfidf word2vec vectorizer", TfidfEmbeddingVectorizer(w2v)), ("tfidf word2vec extra trees", ExtraTreesClassifier(n_estimators=100))])


all_models = [
    ("svc", svc),
    ("svc_tfidf", svc_tfidf),
    ("randomF", randomF), 
    ("randomF_tfidf", randomF_tfidf),
    ("extraT", extraT), 
    ("extraT_tfidf", extraT_tfidf),
    ("etree_w2v", etree_w2v), 
    ("etree_w2v_tfidf", etree_w2v_tfidf)
]
scores = sorted([(name, cross_val_score(model, df['text'], df['product'], cv=kfold).mean()) 
             for name, model in all_models], 
            key=lambda args: -(args[1]))
My guess is that the dataset is very small. Word2Vec won't be able to capture word relationship in the embedding space with limited information. Try to train word2vec on a very large corpus to get a very good word vector before training your classifier might help.