Naives Bayes Text Classifier Confidence Score
First there is a simple theoretical reason why relying on the probability provided by a NB model is not a good idea: what NB predicts is the posterior probability, i.e. the conditional probability $p(C_i|d)$ for every class $C_i$ for a given document $d$. The class $C$ which is predicted is just the one which obtains the highest probability $p(C_i|d)$. The sum of all the posterior probabilities for a particular $d$ is 1 since it's a conditional, and this means that a class is always predicted relatively to the other classes: a high probability for $C$ doesn't mean that the document is very likely to be $C$ in general, it only means that it is much more likely to be $C$ than any of the other classes. If it actually belongs to another class unknown to the model, there's no way the model can represent this.
There is also a practical reason why this is not a great idea: experimentally it is known that NB tends to systematically assign extreme probabilities (and to overfit easily but that's not the point). It does this whether or not the class is correct, and this can easily be checked with your data: if you plot the predicted probability for every instance with a different colour for correct/incorrect prediction, you're likely to see that even the incorrect instances were predicted with a very high probability. It's tempting to interpret the predicted probability as a confidence measure, but it's usually a mistake.
But there is hope: the proper way to deal with potential unknown &quot;classes&quot; (these are not classes technically) is not with regular classification methods, which always rely on a &quot;closed world&quot; assumption (only the classes seen during training exist). There are also open-set classification methods: these are much less common but they are not restricted to the known classes. I think the most common method is probably one-class classification, in particular with one-class SVM.