XGBoost becomes unstable when predicting more than ~300 classes I'm using the Python implementation of XGBoosts (version 0.80) XGBoostClassifier to predict one of a large number of classes.

My feature data consists of a sparse boolean matrix of ~10M rows, ~5k columns, with a density of 0.003. (so about 15 values per row).

My targets consist of 2000+ different classes in a long-tail skewed distribution (~300 classes occur more than 10k times).

For now I'm grouping all of my uncommon classes (&lt;10k) into 'other' so they at least count as negative examples. But ideally, I'd like to be able to use a lot more of the different classes as actual targets.

However, whenever I use, say, 800 different targets (each occurs ~5k+ times), the model becomes unstable: training takes a lot longer, the training loss doesn't seem to converge (I haven't saved the output of this unfortunately) and the predictive quality becomes increasingly poor (predict_proba returning 0.99+ for the wrong classes).

Should I just change some parameter I've overlooked? Am I running out of float precision for my loss? Is it due to sampling in the approx method? Does anyone have any ideas?

My specific classifier:

clf = XGBClassifier(
    n_jobs=8,
    max_depth=7, #arbitrary
    min_child_weight=5, #arbitrary
    subsample=0.5, #arbitrary
    colsample_bytree=0.9, #arbitrary
    tree_method='approx',
    objective='multi:softprob',
    eval_metric='mlogloss',
    silent=False
)
I am affraid that it may not be a problem about tuning the XGB classifier, but about your dataset. 

Even with 10M rows, a density of 0.003 over 5000 rows seems far from enough to get interesting results about 300+ classes. 

I would suggest to start with a way simpler approach to confirm that the modelling approach is okay. That would mean :


Group your variables significantly more (&lt;100), identify and keep the one that appears significant predictors
Group your classes significantly more (&lt;10)
Start with a simpler model (random forest maybe)


Once you do that and get baseline performance metrics, you can try to add more classes / variables / complexity to your model and see if it actually improve.

A one step calibration process, where you dump all your data into a very complex model is rarely the way to go.