Multi-Class Text Classification: Doc2Vec performing very bad compared to Hashing Vector I have a multi-class text classification problem in hand this is similar to product category mapping where we map products to its correct Category based on the text content provided.

I first created a solution with Hashing Vector and SGD classifier with actually gave around ~84% accuracy.

After going through many online content I found that Doc2Vec is the current cutting-edge representation of Document or paragraph in numerical format. 

So I changed my solution to use Doc2Vec method for Feature Engineering but the accuracy got from this is only ~54%. 

Code:

Reading and Cleansing Data

import logging
import datetime
import re
import string
import codecs
import pandas as pd
import numpy as np

from gensim.models.doc2vec import Doc2Vec,TaggedDocument
from sklearn.feature_extraction.text import HashingVectorizer
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score
from sklearn.utils.class_weight import compute_class_weight
from sklearn.metrics import accuracy_score
from sklearn.linear_model import SGDClassifier

import warnings
warnings.filterwarnings("ignore")

#Reading the input/ dataset
data_file = "Consolidated_input_dataset.txt"
data = pd.read_csv(data_file, header = 0, delimiter= "\t", quoting = 3, encoding = "utf8")
data = data.dropna()

#Cleansing the input dataset removing non alphabets 
data['cleansed_desc'] = data.COMMODITY_DESC.str.lower().str.replace('[^a-z]',' ').str.replace('\s+',' ')

#Spliting to list for traing Doc2Vec
data['cleansed_desc_split'] = data.cleansed_desc.str.split()
train_data, test_data, train_label,  test_label = train_test_split(data[["cleansed_desc", "cleansed_desc_split"]], 
                                                                   data[["Label"]], 
                                                                   test_size=0.3, random_state=100, stratify=data.Label)


Hashing Vector:

sgd_model_full = SGDClassifier(loss='modified_huber',  n_jobs=-1, n_iter=8,
        random_state=42, alpha=1e-06, class_weight="balanced", verbose= 2)

sgd_model.fit(train_data.Doc2Vec.tolist(), train_label.Label)

#Predict Output
output_node1_predict = sgd_model.predict(test_data.Doc2Vec.tolist())

print(accuracy_score(test_label.Label, output_node1_predict))


#Train Model
sgd_model_full = SGDClassifier(loss='modified_huber',  n_jobs=-1, n_iter=8,
        random_state=42, alpha=1e-06, class_weight="balanced", verbose= 2)

vectorizer = HashingVectorizer(n_features=90000, ngram_range=(1,3))

vectorizer.fit(train_data.cleansed_desc)
data_features = vectorizer.transform(train_data.cleansed_desc)
sgd_model.fit(data_features, train_label.Label)

#Predict Output
test_features = vectorizer.transform(test_data.cleansed_desc)
output_node1_predict = sgd_model.predict(test_features)

print(accuracy_score(test_label.Label, output_node1_predict))


Output:

84%


Doc2Vec:

#Creating Doc2Vec

data_tagged = train_data.apply(
    lambda r: TaggedDocument(words=r['cleansed_desc_split'], tags=[train_label.loc[r.name].Label]), axis=1)

doc2vec_test = Doc2Vec(dm=0, vector_size=100, negative=5, hs=0, min_count=2, sample=0, epochs=5, workers=8)
doc2vec_test.build_vocab(data_tagged)
doc2vec_test.train(data_tagged, total_examples=doc2vec_test.corpus_count, epochs=doc2vec_test.iter)

train_data["Doc2Vec"] = train_data.cleansed_desc_split.apply(lambda x : doc2vec_test.infer_vector(x))
test_data["Doc2Vec"] = test_data.cleansed_desc_split.apply(lambda x : doc2vec_test.infer_vector(x))

#Train Model
sgd_model_full = SGDClassifier(loss='modified_huber',  n_jobs=-1, n_iter=8,
        random_state=42, alpha=1e-06, class_weight="balanced", verbose= 2)

sgd_model.fit(train_data.Doc2Vec.tolist(), train_label.Label)

#Predict Output
output_node1_predict = sgd_model.predict(test_data.Doc2Vec.tolist())

print(accuracy_score(test_label.Label, output_node1_predict))


output:

54%


Parameters for Doc2 Vec:

Vector Size =100
Window = 10
Epoch=5
min_count=2
Negative=5

Total number of Documents = 3450000+
Vocabulary Size = 46000+
If I read your model correctly, you only performed 5 epochs with the Doc2Vec model.  This is probably not be enough for the network to learn the word embeddings.  Has your loss leveled out after 5 epochs?  Try running it for 50 epochs and see if it makes any difference.  Conceivably, you could need thousands of epochs to achieve a reasonable model.