classifier predicts only one class I have a classification that has to predict three different classes: gcc,icc, clang. The prblem is that if I use a blind test set to do a submission, when I look athe the prediction I have on it I find that most of the predictions are only of one type. So I have the following situation:



My code is the following:

#pakages
import numpy as np
import pandas as pd
import json as j
import re
import nltk
from nltk.tokenize import word_tokenize


from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer

from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import confusion_matrix, classification_report
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier

#for visualizing data
import matplotlib.pyplot as plt
import seaborn as sns; sns.set(font_scale=1.2)

%matplotlib inline

print('Libraries imported.')

dataFrame = pd.read_json('training_dataset.jsonl',lines = True)
dataFrame

dataFrame['opcodes'] = dataFrame['instructions'].apply(lambda x:[i.split()[0] for i in x])

df = dataFrame[["opcodes", "compiler"]].copy()
df

df['opcodes'] = [" ".join(opcode) for opcode in df['opcodes'].values]

from sklearn.feature_extraction.text import TfidfVectorizer 

tfidf_vectorizer=TfidfVectorizer(ngram_range=(2,2),min_df=2,max_df=0.5)


df_x = df['opcodes']


X_all = tfidf_vectorizer.fit_transform(df_x)
y_all = dataFrame['compiler']

X_train, X_test, y_train, y_test = train_test_split(X_all, y_all, 
    test_size=0.2, random_state=15)

from sklearn import svm
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

clf = RandomForestClassifier(n_estimators=100).fit(X_train,y_train)

y_pred = clf.predict(X_test)

acc = clf.score(X_test, y_test)    
print("Accuracy %.3f" %acc)


Now, using a random forest classifier I get an accuracy of 0.89, but the problem is that the model predicts mostly the class gcc, so I guess something is wrong. 

I was almost soing a submission when I noticed this. 

Does this happen because I didn't consider the fact that I have a multiclass classification problem? I really don't understand.

Can somebody please help me? Thank's in advance.

[EDIT] The dataset is balanced.

[EDIT 2] When try to do the prediction on the blind test set I do the following:

test = pd.read_json('test_dataset_blind.jsonl',lines = True)
test

test['instructions'] = [" ".join(opcode) for opcode in 
test['instructions'].values]

tfidf_vectorizer=TfidfVectorizer(ngram_range=(2,2),min_df=2,max_df=0.5)
df_x_new = test['instructions']


X_new = tfidf_vectorizer.fit_transform(df_x)

new_pred_class = clf.predict(X_new)[:3000]
len(new_pred_class)

sub_compiler = pd.DataFrame({'instructions': 
test['instructions'],'compiler':new_pred_class})
sub_compiler.to_csv('sub_compiler.csv')
sub_compiler.head()


[EDIT 4]anctually the problem seems to be on how I select the data to test. Infact, if I do new_pred_class = clf.predict(X_new)[-3000:], I get:


`
so this time mostly the clang are predicted. 

I have 30000 rows in the original dataset and 3000 in the blind test set.

[EDIT 5]So I think I have to reorganize the select the new_pred_class taking random values from clf.predict(X_new)[:3000], so what I mean is that I should take 3000 random values. How could I do that? Thank's again.

EDIT 6] if I do type(X_new)  I have scipy.sparse.csr.csr_matrix
It is a common problem that - with unbalanced classes - some model tends to predict mostly the majority class. You could try to oversample the minority classes. In addition, RF tends to perform weak here. Boosting or NN are often able to recover more details, which can be important to predict the minority classes.

Edit: Okay, now that you clarified that you face a balanced problem, I guess your problem is the classifier. You could try multinominal logit with lasso regulation to „shrink“ irrelevant features (words). You should also consider Boosting or NN (maybe LSTM sequential).