Eliminate low quality predictions in a classification task
An alternative to thresholding the classification probability would be to set a threshold on the ratio between the highest reported probability and the second highest reported probability. For example, a threshold of 2 would be interpretable as: "Only retain classifications where the likelihood of the class assignment is at least twice as likely as the next most likely class."