Why do probabilities sum to one and how can I set optimal threshold level?
Question 1. Why do probabilities sum to 1: Probability theory. Probabilities sum 1 because that’s how we define them. It just so happens that, by forcing them to sum 1, they have an intuitive interpretation and also calculations end up being easier. But this is mere convenience. Probabilities (or, more specifically, probability measures) could have been defined to sum 12 or 100 or whatever number you prefer. It doesn’t really matter.

Further sub-questions:


Its not the data thats exclusive its the classes.
Classifier is able to classify just with lower probability ( I would actually argue that 99999% is to much maybe he is overfitting)


Question 2 Can I set the threshold, like 90%, and conclude if the probability is less than 90% , let the users manually classify the data? Let the data tell you optimal cut-off level/threshold (from the ROC curve, read it up)

def optimal_cutoff(ground_truths: np.array,
                             predictions: np.array) -&gt; float:
        """

        :param ground_truths: array of arrays (matrix) of all ground truths
        :param predictions: array of arrays (matrix) of all predictions
        :return: optimal cut-off level
        """
        optimal_thresholds = []
        for y_train, y_pred in zip(ground_truths, predictions):
            fpr, tpr, threshold = roc_curve(y_train, y_pred)
            optimal_idx = np.argmax(tpr - fpr)
            optimal_thresholds.append(threshold[optimal_idx])

        return sum(optimal_thresholds) / (len(optimal_thresholds))


So you would calculate cut-off on your available GROUND truths. Now if, when you specified optimal cut-off level your metric (lets say f1) falls behind certain level you can tell users to check it up.