Python Text Classification - Data that does not fit into any category I am having a lot of trouble finding any kind of answers to this problem i am facing.
I have a few text classifiers that i am testing out, and they work well for data that does fit into any predefined category, but if I input, lets just say &quot;fhjakdlfsah&quot;, it will still assign it to some category because i guess the predict_proba functionality has to add up to 1 for all of the categories.
Is there something i am missing here? I am having such a hard time finding a solution to this, and I would imagine it is a very common thing to deal with. Right now i am working with gradientboosting from sklearn, and tried wrapping it in a onevsrestclassifier as suggested by others, but it is still having the same thing where all probabilities are adding up to one, and it is getting assigned the highest probability
Basically I am looking for a solution that can say either, yes this fits into one of these categories, or no, this does not fit into any of these categories.
Any help would be greatly appreciated as I am getting quite stuck here
Thanks!
The main assumption in supervised learning is that the training set is a representative sample of the space of all possible instances for the problem.
This is why in the regular classification setting there is simply no way to consider a &quot;not in any class&quot; option: what the model learns is to distinguish between the classes seen in the training set, nothing else. In the standard one-vs-rest multi-class setting It's important to understand that the model doesn't predict whether an instance is likely to belong to &quot;class A&quot; in general, it only predicts whether the instance is more likely to belong to class A compared to any other known class. This is a common misunderstanding, but one-vs-rest is not an answer to the problem of &quot;not in any class&quot;.
There are options to deal with this problem, but one must design the system so that the model can deal with such cases. In general the problem is actually open-set classification: in regular classification the set of possible classes is closed, as mentioned above. The open-set classification setting is much less standard and usually harder, since the model has to make a prediction about something it does not know from its training data. Here are a few options:

Treat the problem as an outlier detection problem: before running the regular classifier, eliminate abnormal instances which don't look like the regular training set cases.
Multi-label classification: in multi-label classification an instance may belong to zero, one or several classes. A binary classifier is trained for every class, and an instance may belong to none of the classes. This is more flexible than multi-class classification, but it's still closed-set classification: for example the model is supposed to have seen examples in the training set which don't belong to any class.
One-class classification is a type of classification where the model learns to identify a single class &quot;in general&quot;, not by contrast to any other class. This is proper open-set classification: if a model is trained for every known class and every model predicts the instance as negative, then the instance doesn't belong to any known class.