XGBoost becomes unstable when predicting more than ~300 classes
I am affraid that it may not be a problem about tuning the XGB classifier, but about your dataset. 

Even with 10M rows, a density of 0.003 over 5000 rows seems far from enough to get interesting results about 300+ classes. 

I would suggest to start with a way simpler approach to confirm that the modelling approach is okay. That would mean :


Group your variables significantly more (&lt;100), identify and keep the one that appears significant predictors
Group your classes significantly more (&lt;10)
Start with a simpler model (random forest maybe)


Once you do that and get baseline performance metrics, you can try to add more classes / variables / complexity to your model and see if it actually improve.

A one step calibration process, where you dump all your data into a very complex model is rarely the way to go.