Help with training XGB
You could e.g. use as objective "multi:softprob" instead of "multi:softmax". The model is identical, but you get a probability for each label instead if the most probable label.  Then you can specify a threshold, e.g. 0.5 and return "unknown" if no probability is above the limit.