Building a multiclass classifier that can handle classes it has never seen? I am given a dataset that has free-form text and a category associated with it. There are 100 different categories and 3000 records for each category. The goal is build a multiclass classification model. I have created a simple neural network with 10,000 input features/words and the results were fairly good (~88%).
The issue I am facing is with an unlabeled dataset that I have, that is missing the category label. This dataset is very large, and has much more than 100 categories. I am only interested in being able to classify unlabeled data for the 100 categories that I have, but I am not sure how to approach this.
One thought I had was to build a Word Embedding model for the labeled data. This model could be used to calculate a document vector for the unlabeled data and find a similar document from the labeled dataset. This would allow me to assign labels to some of the data in the unlabeled dataset. Is there a better way to approaching this problem?
You basically want your model to say - I dont know.
For the test set created out of the original dataset with labels, plot a histogram of the probabilities of the predicted class. Are they generally higher than some threshold say 0.2 ? If so experiment using that as a threshold to flag items from your unlabeled dataset for a manual inspection. The hope is that the probabilities are low because the model is unsure.
The manual review of these flagged items will allow you to verify if this approach even works. If it works, then use that as a labeling strategy for the &quot;I dont know&quot; category and retrain along with this new label.