ROC AUC score is better if test data is imbalanced I have an imbalanced dataset and I'm using XGBoost to do binary classification. I used down sampling together with target and one hot encoding for train data. For test data I once used just the encodings and left it unbalanced and once tried with a balanced test dataset. 

The ROC AUC score was quite higher for the imbalanced test data than the balanced one. How is this possible? I thought for the ROC AUC score there should not be any difference?
ROC normally reports higher scores for imbalanced data sets because it does not take into account false discoveries. For this reason, it is recommended to only use ROC when the positive and negative sets are roughly equal in size. Otherwise, the suggestion is to compute the area under the Precision-Recall Curve instead.

You can read a bit about the reasons here.

https://machinelearningmastery.com/roc-curves-and-precision-recall-curves-for-classification-in-python/


  However, ROC curves can present an overly optimistic view of an algorithm’s performance if there is a large skew in the class distribution. […] Precision-Recall (PR) curves, often used in Information Retrieval , have been cited as an alternative to ROC curves for tasks with a large skew in the class distribution.
  
  — The Relationship Between Precision-Recall and ROC Curves, 2006.