XG Boost result interpretation for unbalanced datasets (Accuracy & AUCROC) My dataset is of shape – 5621*8 (binary classification)

Label/target : Success (4324, 77 %) &amp; Not success (1297, 23 %)
(success and Not success were been equally important for my prediction i.e, TP &amp; TN)


I split my data into 3 (Train, Validate, test)

For train &amp; Validate i perform 10 fold CV.

Test is the seperate data, which I evaluate for each folds


I tune my scale_pos_weight ranging between 5 to 80, and

Finally I fixed my values as 75 since I got average higher accuracy rate for my Test set (79 %) for those 10 folds
But, If i check my average auc_roc metrics it is very poor, i.e only 50 % for all 10 folds.

If i did not tune scale_pos_weight my avg.accuracy drops to 50% &amp; my avg auc_roc increases to 70 %.
How can I interpret from the above results between AUCROC &amp; Accuracy in this situation?
What might be the problem in my case?
With Success already being the larger class, you probably shouldn't be using a scale_pos_weight larger than one: you want to scale the positive class's contribution to the loss function down rather than up.
I suspect that's what's happening in the first case.  With scale_pos_weight=75, the model ends up basically only caring about the positive class, predicts everyone is in the positive class, and so your accuracy is just a little better than the 77% baseline you'd expect with that strategy.  With that motivation, it's not too surprising the AUC is poor, although I wouldn't have expected a drop all the way to the 50% baseline...
If you don't use scale_pos_weight (you said &quot;if I did not tune&quot;, but does that mean you left it at the default 1?), then the model performs better in rank-ordering (AUC=70%), but not so well in the hard classification.  You might want to tweak the prediction threshold here; there's probably a different threshold that will perform better for accuracy score.  You could also try scale_pos_weight=0.25 or so; that should make the default cutoff better, hopefully with little effect on AUC?