Oversampling possible improvement I am currently solving a classification problem for an imbalanced data set (approximately 17% of the minority class). I split the data using a stratified k-fold split from sklearn (Stratified shuffle split) after which I oversample the train data, using ADASYN, and fit the oversampled train data (approximately 250k+ instances after oversampling) to gradient boosting classifier. Oversampling has a huge effect on the performance, recall measure improves from 7% to 75%. Is this possible? If not, any ideas what could be going wrong?

My main question about this is could this improvement be possible?
The evaluation of an imbalanced dataset should be done with more than one single metric since you need to evaluate the performance on both the majority and the minority classes.

In the case that is mentioned the performance is evaluated only with recall (the true-positive rate). It could be that the model is now focusing on the oversampled minority class (especially if it's the new majority class) and having a weaker performance on the original majority class, and I assume that the precision or the false-negative rate were decreased.

There are several evaluation metrics that are considering the performance of both classes that are not sensitive to the classes imbalance - F1 score, Geometric mean, and the most common the ROC/PR AUC