Xgboost in R: Dealing with extreme class imbalance My data has an extreme class imbalance - 99.7% is 0's and 0.2% is 1's and almost all the predictor variables (6 out of 7) are categorical. I trained an xgboost classifier after performing an upsampling (using ROSE in R). Evaluated the model on 20% test data and the following is the result,


Recall, as you can see, is not great. Additionally, if I run the model on a completely different test data (meaning not a chunk of the original data), then the recall value drops to exactly 50%. Is this happening because it is overfitting? Can someone please let me know how I can better this model, considering the extreme class imbalance? 

I'm a newbie in the field. Any help would be much appreciated. Thanks!
The drop from 64 to 50 doesn't seem like overfitting, if your test set was properly held back from model development.  (And from the confusion matrix it looks like you haven't made the fairly common mistake of upsampling before splitting.)

If the "completely different test data" is not identically distributed to the training data, that can account for the discrepancy.  There aren't universally good ways to account for failure of the iid data assumption; you could try to underfit the model more, on the assumption that there's some underlying large trend that's constant across datasets and the model is finding (and overfitting to) some additional, more localized, trend in the train/validation set.