how to identify the unknown class in machine learning?
I have come across a very similar problem lately. It is a case of novelty/anomaly detection as you have mentioned.

There are broadly 2 "working" solutions to these type of problems:


You can check the softmax-posterior probability value of the output layer for all classes and threshold it(say 0.5). If the probability is &lt; 0.5 for all other classes, then definitely this test case is an outlier(our unknown class in this case).
Adding a Garbage class and training it. But, collecting/randomly generating the features for this unknown class may sometimes become tedious and difficult. It is dependent on the application and on the number of classes. If you have smaller number of classes(&lt;10), then you could try this too.


Best would be to try both and see which one best fits the purpose of your application and which increases your prediction accuracy positively.