Predict_proba on a binary classification problem
sklearn provides us with two methods to calibrate a probabilistic classifier via their CalibratedClassifierCV class; one using Platt's scaling (sigmoid) and one using isotonic regression.

Another method is by using a Venn-Abers predictor, which is not implemented in sklearn, however you can find a custom implementation here.

As for imbalanced datasets, make sure to exploit the class_weights argument of an sklearn classifier, to increase the weight of under-represented classes.