Accuracy reduces drastically when using TruncatedSVD with hashingvector I have around 0.8 million product description with categories. There are around 280 categories. I want to train a model with given dataset so that in future I can predict Category for the given product description. Since the dataset was large I was not able to able TF-IDF on that data it throws MemoryError. 

I found that Hashingvector was desirable when dealing with large data. But when Hashingvector was applied I found it produced data with 1048576 features. It took around 1 hour to train and SGD model and produced 78% accuracy.

Code:

import pandas as pd
from sklearn.feature_extraction.text import HashingVectorizer
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.decomposition import TruncatedSVD
from sklearn.linear_model import SGDClassifier
from sklearn.calibration import CalibratedClassifierCV

data_file = "Product_description.txt"
#Reading the input/ dataset
data = pd.read_csv(data_file, header = 0, delimiter= "\t", quoting = 3, encoding = "utf8")
data = data.dropna()

train_data, test_data, train_label,  test_label = train_test_split(data.Product_description, data.Category, 
                                                                   test_size=0.3, random_state=100, stratify=data.Category)

sgd_model = SGDClassifier(loss='hinge', n_iter=20, class_weight="balanced", n_jobs=-1,
                          random_state=42, alpha=1e-06, verbose=1)
vectorizer = HashingVectorizer(ngram_range=(1,3))
data_features = vectorizer.fit_transform(train_data.Product_description)
sgd_model.fit(data_features, train_label)
test_data_feature = vectorizer.transform(test_data.Product_Combined_Cleansed)
Output_predict = sgd_model.predict(test_data_feature)
print(accuracy_score(test_label, Output_predict))


Output:

Accuracy 77.01%


Since the dimension was high I thought reducing the dimension would increase the accuracy and reduce the training time. I used TrancatedSVD to reduce the dimension but this drastically reduced the prediction accuracy but reduced the training time to 10 minutes.

Code2:

from sklearn.decomposition import TruncatedSVD
clf = TruncatedSVD(100)
clf.fit(data_features)


Output:

Accuracy 14%


Edit:

When I tried TruncatedSVD with 1000 as limit it throws memory error so only I opted to use 100 as the limit.

It is stated that reducing n_features on HashingVector will cause collisions there can be collisions: distinct tokens can be mapped to the same feature index. However, in practice, this is rarely an issue if n_features is large enough (e.g. 2 ** 18 for text classification problems) in scikit site.

I got optimum accuracy when I used ngram between 1 to 3 so only used that.
Some possible directions:


when you're unsure how your BoW model works check different n-gram ranges (did you even check how it works with only unigrams?)
TruncatedSVD by default only iterates 5 times (n_iter), so you could increase that
It's not at all surprising to get such drastic change of classification quality for such low dimensionality. Some questions you could ask are why 100 dimensions? Does TSVD work 1000 dimensions or something like that? What is the reconstruction error (how does TSVD's explained_variance_ratio_ look like)?
HashingVectorizer also has parameter that can control number of features, namely n_features.