XGBoost: How to set the probability threshold for multi class classification
You don't set it in xgboost. Its job is to return probabilities in predict_proba. predict does the logical thing and tells you the most likely class.

If you want to interpret the probabilities differently, you'd have to write code to do so. It depends on what "does not much differ" means. For example if you simply mean "the most likely class is has probability &lt; 0.5", in numpy it's something like np.argwhere(probabilities.max(axis=1) &lt; 0.5). Those rows are what are undetermined. For the rest the prediction is the argmax of the row.

Of course, that's not the only definition of when it's undetermined, so you need to figure out what the rule is that you're meant to implement. You could base it on entropy too, not simply the max class probability.