Probability calibration is worsening my model performance
The probability calibration is just stacking a logistic or isotonic regression on top of the base classifier.  The default is logistic, and since the sigmoid is a strictly increasing function, the rank-ordering of samples will be unaffected, and so AUC should not change at all.

(With isotonic regression, it's actually piecewise-constant, so within a span where the function is constant, all samples will have their scores made equal, and so your ROC curve will become more coarse, which would affect the AUC; but these effects ought to be small, as long as the isotonic fit produces sufficiently many/short constant segments.)

Also, gradient boosting like XGBoost produces scores biased toward the extremes, not away from them like random forests, so the logistic calibration is unlikely to work out well.